Definition of the line

Benchmarking is the process of comparing the performance of a given system or process against a set of standards or a comparable system. Benchmarking requires a set of metrics that can accurately reflect the capabilities and potential of the system under examination. In rehabilitation robotics, common metrics are accuracy, speed, stability, power, weight, energy consumption, range of motion, donning and doffing time, user comfort, etc… The results of the benchmarking analysis can be used to set performance targets, improve the efficiency of R&D processes, predict the effects of different solutions, or verify safety. Particularly important in benchmarking are unified datasets and standard experimental protocols to compare systems on a common ground of tasks and domains. Benchmarking is normally conducted in the laboratory to assure measurability and reproducibility. However, lab tests should be complemented by field tests to verify the ability of benchmarks to predict real world performance.

History/expertise of the group in this line

In our group, we apply benchmarking in two main areas:

  • Human locomotion. Our goal here is to characterize bipedal locomotion over a wide range of terrain conditions. Our method is to design specific testbeds and protocols that can reproduce natural-like environments and create large datasets on humans (both healthy and patients) moving in such environments.
  • Human-exoskeleton interaction. Our goal here is to assess and predict the transmission of forces between an exoskeleton and an individual during a wide range of operative conditions. Our method is based on the use of physical replicas of human body (test dummies) in combination with digital twins.

Related Projects

The group has coordinated different European projects in this field. In the EUROBENCH Project, we developed the first benchmarking framework for bipedal systems, including exoskeleton, prostheses and humanoid robots, and built the first testing facilities for these types of technologies. In the EXOSAFE and SALOEXO projects, we explored the physical interactions between humans and exoskeletons, developing methodologies to test the transmission of forces by using physical replicas of the human limbs and AI methods to predict user’s comfort. The group has extended these benchmarking approaches in the field of Parkinson’s Disease diagnosis, with two national projects, NEUROMARK and AI 4 HEALTHY AGING. The basic idea behind them is to assess the motor activity of patients under a multidimensional perspective, by exposing them to multiple motor tasks under different challenging conditions, using AI to identify good biomarkers of the evolution of symptoms.

Relevant papers

Torres-Pardo et al., Legged locomotion over irregular terrains: state of the art of human and robot performance, Bioinspiration & Biomimetics, 2022. 4.

Longatelli et al., A unified scheme for the benchmarking of upper limb functions in neurological disorders, Journal of NeuroEngineering and Rehabilitation 19(1): pp 1-20, 2022. 2.

Remazeilles et al. Making Bipedal Robot Experiments Reproducible and Comparable: The Eurobench Software Approach, Frontiers in Robotics and AI, 2022. 7.

Torricelli et al., Benchmarking Wearable Robots: Challenges and Recommendations From Functional, User Experience, and Methodological Perspectives. Frontiers in Robotics and AI. 7. 168. 2020.

Pinto-Fernández et al., Performance evaluation of lower limb exoskeletons: a systematic review. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 28. 7. 1573-1583. 2020.

Torricelli et al., A subject-specific kinematic model to predict human motion in exoskeleton-assisted gait. Front. Neurorobot.. 2018. 10.3389/fnbot.2018.00018.

Torricelli et al., “Benchmarking Bipedal Locomotion: A Unified Scheme for Humanoids, Wearable Robots, and Humans,” in IEEE Robotics & Automation Magazine, vol. 22, no. 3, pp. 103-115, Sept. 2015.

Main researchers

Dr. Diego Torricelli
Dr. David Rodríguez