A2 Refereed review article in a scientific journal
A review on deep learning for vision-based hand detection, hand segmentation and hand gesture recognition in human-robot interaction
Authors: Jalayer, Reza; Jalayer, Masoud; Orsenigo, Carlotta; Tomizuka, Masayoshi
Publisher: PERGAMON-ELSEVIER SCIENCE LTD
Publication year: 2026
Journal: Robotics and Computer-Integrated Manufacturing
Article number: 103110
Volume: 97
ISSN: 0736-5845
eISSN: 1879-2537
DOI: https://doi.org/10.1016/j.rcim.2025.103110
Web address : https://doi.org/10.1016/j.rcim.2025.103110
Self-archived copy’s web address: https://research.utu.fi/converis/portal/detail/Publication/500332936
Hand-based analysis, including hand detection, segmentation, and gesture recognition, plays a pivotal role in enabling natural and intuitive human-robot interaction (HRI). Recent advances in vision-based deep learning (DL) have significantly improved robots' ability to interpret hand cues across diverse settings. However, previous reviews have not addressed all three tasks collectively or focused on recent DL architectures. Filling this gap, we review recent studies at the intersection of DL and hand-based interaction in HRI. We structure the literature around three core tasks, i.e. hand detection, segmentation, and gesture recognition, highlighting DL models, dataset characteristics, evaluation metrics, and key challenges for each. We further examine the application of these models across industrial, assistive, social, aerial, and space robotics domains. We identify the dominant role of Convolutional and Recurrent Neural Networks (CNNs and RNNs), as well as emerging approaches such as attention-based models (Transformers), uncertainty-aware models, Graph Neural Networks (GNNs), and foundation models, i.e. Vision-Language Models (VLMs) and Large Language Models (LLMs). Our analysis reveals gaps, including the scarcity of HRI-specific datasets, underrepresentation of multi-hand and multi-user scenarios, limited use of RGBD and multi-modal inputs, weak cross-dataset generalization, and inconsistent real-time benchmarking. Dynamic and long-range gestures, multi-view setups, and context-aware understanding also remain relatively underexplored. Despite these limitations, promising directions have emerged, such as multi-modal fusion, use of foundation models for intent reasoning, and the development of lightweight architectures for deployment. This review offers a consolidated foundation to support future research on robust and context-aware DL systems for hand-centric HRI.
Downloadable publication This is an electronic reprint of the original article. |  |
Funding information in the publication:
The present study has been developed within the HumanTech Project, which is financed by the Italian Ministry of University and Research (MUR) for the 2023–2027 period as part of the ministerial initiative “Departments of Excellence” (L. 232/2016).
