Project

Back to overview

Myo-Electricity, Gaze and Artificial Intelligence for Neurocognitive Examination and Prosthetics

English title Myo-Electricity, Gaze and Artificial Intelligence for Neurocognitive Examination and Prosthetics
Applicant Müller Henning
Number 160837
Funding scheme Sinergia
Research institution Institut Informatique de gestion HES-SO Valais
Institution of higher education University of Applied Sciences and Arts Western Switzerland - HES-SO
Main discipline Information Technology
Start/End 01.02.2016 - 31.07.2019
Approved amount 931'441.00
Show all

All Disciplines (2)

Discipline
Information Technology
Neurology, Psychiatry

Keywords (5)

artificial intelligence; prosthetics; gaze analysis; neurocognitive examination; electro myography

Lay Summary (Italian)

Lead
Con MEGANE PRO, ci proponiamo di portare la ricerca nell'ambito del controllo di protesi della mano ad un livello più avanzato, ossia la comprensione degli effetti neurocognitivi e neurologici dell'amputazione ed il miglioramento delle possibilità di controllo delle protesi. In questo modo, questo progetto può migliorare lo stato dell'arte nella protesica della mano e allo stesso tempo migliorare l'esito clinico dei pazienti.MEGANE PRO è composto da tre sottoprogetti che collaboreranno attivamente:1. Acquisizione, cura e condivisione di Dati (HES-SO Valais, Sierre)2. Analisi neuro-cognitive (Ospedale Universitario di Zurigo)3. Controllo adattivo multi-modale di protesi della mano (Università La Sapienza, Roma, Italia)
Lay summary
Con MEGANE PRO, ci proponiamo di portare la ricerca nell'ambito del controllo di protesi della mano ad un livello più avanzato, ossia la comprensione degli effetti neurocognitivi e neurologici dell'amputazione ed il miglioramento delle possibilità di controllo delle protesi. In questo modo, questo progetto può migliorare lo stato dell'arte nella protesica della mano e allo stesso tempo migliorare l'esito clinico dei pazienti. MEGANE PRO è composto da tre sottoprogetti che collaboreranno attivamente: 1. Acquisizione, cura e condivisione di Dati (HES-SO Valais, Sierre) 2. Analisi neuro-cognitive (Ospedale Universitario di Zurigo) 3. Controllo adattivo multi-modale di protesi della mano (Università La Sapienza, Roma, Italia)
Direct link to Lay Summary Last update: 20.07.2015

Lay Summary (English)

Lead
With MEGANE PRO, we aim to bring the research on prosthesis control of hand amputated subjects to its next level, i.e. to better understand the neurologic and neurocognitive effects of amputation on the persons and to strongly improve robotic prosthesis control possibilities by hand amputated subjects. Thus, this project can improve the state of the art in hand prosthetics and also improve the clinical outcome of the patients.MEGANE PRO is composed of three subprojects that collaborate continuously.1. Data Acquisition, Curation and Sharing (Information System Institute, HES-SO Valais, Sierre)2. Neuro-Cognitive Analyses (University Hospital, Zurich)3. Multi-Modal Adaptive Control of Prosthetic Hands (La Sapienza University, Rome, Italy)
Lay summary
With MEGANE PRO, we aim to bring the research on prosthesis control of hand amputated subjects to its next level, i.e. to better understand the neurologic and neurocognitive effects of amputation on the persons and to strongly improve robotic prosthesis control possibilities by hand amputated subjects. Thus, this project can improve the state of the art in hand prosthetics and also improve the clinical outcome of the patients.

MEGANE PRO is composed of three subprojects that collaborate continuously.

1. Data Acquisition, Curation and Sharing (Information System Institute, HES-SO Valais, Sierre)
2. Neuro-Cognitive Analyses (University Hospital, Zurich)
3. Multi-Modal Adaptive Control of Prosthetic Hands (La Sapienza University, Rome, Italy)
Direct link to Lay Summary Last update: 20.07.2015

Responsible applicant and co-applicants

Employees

Publications

Publication
Kinematic synergies of hand grasps: a comprehensive study on a large publicly available dataset
Jarque-Bou Néstor J., Scano Alessandro, Atzori Manfredo, Müller Henning (2019), Kinematic synergies of hand grasps: a comprehensive study on a large publicly available dataset, in Journal of NeuroEngineering and Rehabilitation, 16(1), 63-63.
A quantitative taxonomy of human hand grasps
Stival Francesca, Michieletto Stefano, Cognolato Matteo, Pagello Enrico, Müller Henning, Atzori Manfredo (2019), A quantitative taxonomy of human hand grasps, in Journal of NeuroEngineering and Rehabilitation, 16(1), 28-28.
PaWFE: Fast Signal Feature Extraction Using Parallel Time Windows
Atzori Manfredo, Müller Henning (2019), PaWFE: Fast Signal Feature Extraction Using Parallel Time Windows, in Frontiers in Neurorobotics, 13, 74.
Natural Myocontrol in a Realistic Setting: a Comparison Between Static and Dynamic Data Acquisition
Gigli Andrea, Gijsberts Arjan, Castellini Claudio (2019), Natural Myocontrol in a Realistic Setting: a Comparison Between Static and Dynamic Data Acquisition, in 2019 IEEE 16th International Conference on Rehabilitation Robotics (ICORR), Toronto, ON, CanadaIEEE, Toronto, Canada.
Analyzing the trade-off between training session time and performance in myoelectric hand gesture recognition during upper limb movement.
Cognolato Matteo, Brigato Lorenzo, Dicente Cid Yashin, Atzori Manfredo, Müller Henning (2019), Analyzing the trade-off between training session time and performance in myoelectric hand gesture recognition during upper limb movement., in ICORR 2019, Toronto (Canada)IEEE, NA.
The Difficulty of Recognizing Grasps from Semg During Activities of Daily Living"
Gregori Valentina, Caputo Barbara, Gijsberts Arjan (2018), The Difficulty of Recognizing Grasps from Semg During Activities of Daily Living", in Biorob, IEEE, Twente (Netherlands).
Hand Gesture Classification in Transradial Amputees Using the Myo Armband Classifier* This work was partially supported by the Swiss National Science Foundation Sinergia project # 410160837 MeganePro.
Cognolato M., Atzori M., Faccio D., Tiengo C., Bassette F., Gassert R., Muller H. (2018), Hand Gesture Classification in Transradial Amputees Using the Myo Armband Classifier* This work was partially supported by the Swiss National Science Foundation Sinergia project # 410160837 MeganePro., in 2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob), EnschedeIEEE, NA.
Visual Cues to Improve Myoelectric Control of Upper Limb Prostheses
Gigli Andrea, Gregori Valentina, Cognolato Matteo, Atzori Manfredo, Gijsberts Arjan (2018), Visual Cues to Improve Myoelectric Control of Upper Limb Prostheses, in 2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob), EnschedeIEEE, NA.
Head-mounted eye gaze tracking devices: An overview of modern devices and recent advances.
CognolatoMatteo, AtzoriManfredo, MüllerHenning (2018), Head-mounted eye gaze tracking devices: An overview of modern devices and recent advances., in Journal of Rehabilitation and Assistive Technologies Engineering, NA-NA.
Apparent motion perception in lower limb amputees with ph antom sensations: “Obstacle Shunning” and "Obstacle Tolerance"
Saetta Gianluca, Grond Ilva, Brugger Peter, Lenggenhager Bigna, Tsay Anthony, Giumarra Melita (2018), Apparent motion perception in lower limb amputees with ph antom sensations: “Obstacle Shunning” and "Obstacle Tolerance", in Cortex.
Apparent motion perception in lower limb amputees with ph antom sensations: “Obstacle Shunning” and “Obstacle Tolerance”
Saetta Gianluca, Grond Ilva, Brugger Peter, Lenggenhager Bigna, Tsay Anthony, Giumarra Melita (2018), Apparent motion perception in lower limb amputees with ph antom sensations: “Obstacle Shunning” and “Obstacle Tolerance”, in European Annual Congress of Cognitive Neuroscience, unknown, unknown.
Variability of sEMG classification accuracy in different hand movements for intact and hand amputated subjects
Atzori Manfredo, Tiengo Cesare, Coppola M., Cognolato Matteo, Faccio D., Bassetto Franco, Müller Henning (2018), Variability of sEMG classification accuracy in different hand movements for intact and hand amputated subjects, in Innovations in Amputation Surgery and Prosthetic Technologies (IASPT), unknown, Conference.
Adaptive learning to speed-up control of prosthetic hands: A few things everybody should know.
Gregori Valentina, Gijsberts Arjan, Caputo Barbara (2017), Adaptive learning to speed-up control of prosthetic hands: A few things everybody should know., in ICORR, unknown, unknown.
Analysis of the repeatability of grasp recognition for hand robotic prosthesis control based on sEMG data
Palermo Francesca, Cognolato Matteo, Gijsberts Arjan, Caputo Barbara, Müller Henning, Atzori Manfredo (2017), Analysis of the repeatability of grasp recognition for hand robotic prosthesis control based on sEMG data, in IEEE International Conference on Rehabilitation Robotics, LondonIEEE, Conference.
Comparison of Six Electromyography Acquisition Setups on Hand Movement Classification Tasks
Pizzolato Stefano, Tagliapietra Luca, Cognolato Matteo, Reggiani Monica, Müller Henning, Atzori Manfredo (2017), Comparison of Six Electromyography Acquisition Setups on Hand Movement Classification Tasks, in Plos One, e0186132.
Megane Pro: myo-electricity, visual and gaze tracking integration as a resource for dexterous hand prosthetics
Giordaniello Francesca, Cognolato Matteo, Graziani Mara, Gijsberts Arjan, Gregori Valentina, Saetta Gianluca, Hager Anne-Gabrielle Mittaz, Tiengo Cesare, Bassetto Franco, Brugger Peter, Caputo Barbara, Müller Henning, Atzori Manfredo (2017), Megane Pro: myo-electricity, visual and gaze tracking integration as a resource for dexterous hand prosthetics, in IEEE International Conference on Rehabilitation Robotics, IEEE, Conference.
Semi-automatic training of an object recognition system in scene camera data using gaze tracking and accelerometers
Cognolato Matteo, Graziani Mara, Giordaniello Francesca, Saetta Gianluca, Bassetto Franco, Brugger Peter, Caputo Barbara, Müller Henning, Atzori Manfredo (2017), Semi-automatic training of an object recognition system in scene camera data using gaze tracking and accelerometers, in International Conference on Computer Vision Systems (ICVS), IEEE, Conference.
The effect of change in motor cortex excitability on the apparent motion perception of human body parts: a transcranial direct current stimulation (tDCS) study
Saetta Gianluca, Inauen Malin, Lenggenhager Bigna, Brugger Peter (2017), The effect of change in motor cortex excitability on the apparent motion perception of human body parts: a transcranial direct current stimulation (tDCS) study, in 3rd Urobody Meeting, unknown, unknown.
The effect of change in motor cortex excitability on the apparent motion perception of human body parts: a transcranial direct current stimulation (tDCS) study
Saetta Gianluca, Inauen Malin, Lenggenhager Bigna, Brugger Peter (2017), The effect of change in motor cortex excitability on the apparent motion perception of human body parts: a transcranial direct current stimulation (tDCS) study, in 27th Symposium ZHIP , unknown, unknown.
“Obstacle Shunning”. A Behavioral and psychophysiological study in lower limb amputees with phantom sensations
Saetta G, Grond I, Brugger P, Lenggenhager B, Tsay A, Giummarra M (2016), “Obstacle Shunning”. A Behavioral and psychophysiological study in lower limb amputees with phantom sensations, in Users’ Body Experience and Human-Machine Interfaces in (Assistive) Robotics – URoBody meeting on “bo, Olten, Switzerlandunknown, unknown.
Bodily obstacles in mental imagery. Specific perspective-dependent effects of the apparent motion perception of an actors' body parts
Saetta Gianluca, Atzori Manfredo, Caputo Barbara, Mü}ller Henning, Lenggenhagger Bigna, Brugger Peter (2016), Bodily obstacles in mental imagery. Specific perspective-dependent effects of the apparent motion perception of an actors' body parts, in 12th Symposium of the Zurich Center for Integrative Human Physiology (ZIHP), Zurich, Switzerlandunknown, unknown.
Deep Learning with Convolutional Neural Networks Applied to Electromyography Data: A Resource for the Classification of Movements for Prosthetic Hands
Atzori Manfredo, Cognolato Matteo, Müller Henning (2016), Deep Learning with Convolutional Neural Networks Applied to Electromyography Data: A Resource for the Classification of Movements for Prosthetic Hands, in Frontiers in Neurorobotics, 10, x.
Electromyography for Hand Prosthetics Demo
Atzori Manfredo, Schaer Roger, Cognolato Matteo, Mü}ller Henning (2016), Electromyography for Hand Prosthetics Demo, in XXVIII Conférence francophone sur l'Interaction Homme-Machine, Fribourg (Switzerland)ACM, Fribourg.
Obstacles to mental imagery: How bodily constraints affect apparent motion perception of human body parts. A proposed experiment.
Saetta Gianluca, Rothacher Yannick, Cognolato Matteo, Atzori Manfredo, Caputo Barbara, Müller Henning, Lenggenhagger Bigna, Brugger Peter (2016), Obstacles to mental imagery: How bodily constraints affect apparent motion perception of human body parts. A proposed experiment., in International Brain-Computer Interface (BCI) Meeting, Darmstadtunknown, unknown.
Sensorimotor representations underlying the apparent motion perception of human body parts. Evidence from a participant with xenomelia.
Saetta G, Lenggenhager B, Müller H, Brugger P (2016), Sensorimotor representations underlying the apparent motion perception of human body parts. Evidence from a participant with xenomelia., in XXXV European Workshop on Cognitive Neuropsychology, Bressanone, Italyunknown, unknown.

Datasets

Megane Pro dataset

Author Cognolato, Matteo
Persistent Identifier (PID) To be defined
Repository MeganePro
Abstract
Dataset and associated publications currently under review.

Collaboration

Group / person Country
Types of collaboration
Prof. Franco Bassetto, University of Padova, Clinic of Plastic Surgery Italy (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
- Exchange of personnel
Prof. Nicola Petrone, University of Padova Italy (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
- Research Infrastructure
- Exchange of personnel
- Industry/business/other use-inspired collaboration
Dr. Bigna Lenggenhagger , University of Zurich Switzerland (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
Monash University Australia (Oceania)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
DLR-German Aerospace Center Germany (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
- Research Infrastructure
- Exchange of personnel
Prof. Anne-Gabrielle Mittaz Hager, HES-SO, Physiotherapy section Switzerland (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Exchange of personnel

Scientific events

Active participation

Title Type of contribution Title of article or contribution Date Place Persons involved
IEEE International Conference on Rehabilitation Robotics Poster Megane Pro: myo-electricity, visual and gaze tracking integration as a resource for dexterous hand prosthetics. 17.07.2017 London, Great Britain and Northern Ireland Cognolato Matteo; Gregori Valentina; Saetta Gianluca; Atzori Manfredo; Müller Henning; Gijsberts Arjan; Brugger Peter; Caputo Barbara;
IEEE International Conference on Rehabilitation Robotics Poster Analysis of the repeatability of grasp recognition for hand robotic prosthesis control based on sEMG data. 17.07.2017 London, Great Britain and Northern Ireland Caputo Barbara; Müller Henning; Atzori Manfredo; Gregori Valentina; Gijsberts Arjan;
IEEE International Conference on Computer Vision Systems (ICVS) Poster Semi-automatic training of an object recognition system in scene camera data using gaze tracking and accelerometers 10.07.2017 Shenzen, China Atzori Manfredo; Brugger Peter; Müller Henning; Cognolato Matteo;
Tact-Hand project project meeting Talk given at a conference Megane Pro 11.05.2017 Rome, Italy Gijsberts Arjan;
XXXV European Workshop on Cognitive Neuropsychology. Talk given at a conference Sensorimotor representations underlying the apparent motion perception of human body parts. Evidence from a participant with xenomelia. 22.01.2017 Bressanone, Italy Saetta Gianluca; Müller Henning; Brugger Peter;
Users’ Body Experience and Human-Machine Interfaces in (Assistive) Robotics – URoBody meeting on “body experience” Talk given at a conference “Obstacle Shunning”. A Behavioral and psychophysiological study in lower limb amputees with phantom sensations. 14.11.2016 Olten, Switzerland Brugger Peter; Saetta Gianluca;
ACM, XXVIII Conférence francophone sur l'Interaction Homme-Machine Poster lectromyography for Hand Prosthetics Demo 26.10.2016 Fribourg, Switzerland Müller Henning; Atzori Manfredo; Cognolato Matteo;
Cybathlon Symposium Poster From NinaPro to MeganePro towards the natural control of myoelectric prosthetic hand 06.10.2016 Zurich, Switzerland Cognolato Matteo; Brugger Peter; Müller Henning; Atzori Manfredo; Caputo Barbara;
12th Symposium of the Zurich Center for Integrative Human Physiology (ZIHP) Poster Bodily obstacles in mental imagery. Specific perspective-dependent effects of the apparent motion perception of an actors’ body parts. 26.08.2016 Zurich, Switzerland Caputo Barbara; Atzori Manfredo; Saetta Gianluca; Müller Henning; Brugger Peter;
Users’ Body Experience and Human-Machine Interfaces in (Assistive) Robotics - URoBody kick off meeting. Darmstadt, Germany Talk given at a conference Obstacles to mental imagery: How bodily constraints affect apparent motion perception of human body parts. A proposed experiment. 13.06.2016 Darmstadt, Germany Atzori Manfredo; Saetta Gianluca; Caputo Barbara; Brugger Peter; Müller Henning;


Self-organised

Title Date Place

Associated projects

Number Title Start Funding scheme
132700 Non-Invasive Adaptive Hand Prosthetics (NINAPRO) 01.01.2011 Sinergia
181383 "Body Representation and Apparent motion perception of human body parts. A systematic Transcranial Magnetic Stimulation and Behavioural Investigation" 01.08.2019 Doc.Mobility

Abstract

Bilateral and monolateral hand amputated subjects suffer from strong functional deficits due to their impairment. Surface Electromyography (sEMG) currently gives some control capabilities but these are limited, often not natural and usually require long training times. The application of modern machine learning techniques to analyse sEMG activity related to natural movements seems promising but it is far from practice due to two main aspects. First, the effects of the amputation on the nervous system of the subjects are not fully clear; second, there is a strong lack of accuracy in the movement classication accuracy and a few wrong movements can have important negative effects. In recent work, we started to improve this situation through the establishment of a benchmark database of sEMG data for hand movements (htttp://ninapro.hevs.ch/) , which was welcomed with enthusiasm by the scientic community. Other recent scientic papers show that the combination of visual data and electromyography can strongly extend the capabilities of dexterous prostheses.With MEGANE PRO, we aim to bring the research in this field to its next step, i.e. to better understand the neurologic and neurocognitive effects of amputation on the persons and to strongly improve robotic prosthesis control possibilities by hand amputated subjects. Thus, this project could improve the state of the art in hand prosthetics and also improve the clinical outcome of the patients (e.g., by respecting the individual phantom limb phenomenology). We aim to pursue this objective along four collaborating approaches that are the sub-projects.Subproject 1: data acquisition, curation and sharing (Information Systems Institute, HES-SO, Sierre) - The subproject consists of three main parts in which all teams collaborate. During the first part, we create the acquisition setup and protocol that is based on the fusion between sEMG and gaze protocols. During the second, we recruit hand amputated subjects mainly in Switzerland (and potentially abroad) and we perform data acquistions. During the third, we perform data curation and release the data on the Ninapro websiteSubproject 2: neuro-cognitive analyses (University Hospital, Zurich) - Subproject 2 will improve present-day understanding of hand-grasp-associated predictive eye movements in natural settings (almost the entire literature with normally-limbed subjects has focused on handling artificial laboratory situations). The subproject consists of three main parts. First, we investigate eye movements during simple grasps and manipulations of objects in natural situations. Second, we study how far eye movements during grasping are predictive of situational demands (e.g. object characteristics or task goals). Third, we explore individual differences in participants gaze behavior during imagined object manipulations.Subproject 3: unimodal adaptive control of prosthetic hands (IDIAP Research Center, Martigny) - This subproject focuses on the development of adaptive learning algorithms capable to exploit prior knowledge gathered by the user as well as from intact subjects. IDIAP and UNIROMA1 will mainly collaborate in it using data acquired by the HES-SO. Subproject 4: multi-modal adaptive control of prosthetic hands (La Sapienza University, Rome, UNIROMA1, Italy) - The overall goal of this subproject is to develop a coherent framework of learning algorithms able to significantly advance the state of the art in sEMG controlled prostheses in terms of control, stability and dexterity. The prosthesis controllers will be enriched by autonomous decision making that operate through the analysis of multimodal data. Research is inspired and where suitable driven by the results obtained in subprojects 2 and 3, and all algorithms will be tested on the data collected in subproject 1.
-