Enhanced Human-Robot Teaming Through Attention Multi Convolutional Neural Network-Based Multi-Modal Sensor Fusion for Hand Gesture Recognition and Orientation Control
Files
Date
2025-01-07
Contributor
Advisor
Department
Instructor
Depositor
Speaker
Researcher
Consultant
Interviewer
Narrator
Transcriber
Annotator
Journal Title
Journal ISSN
Volume Title
Publisher
Volume
Number/Issue
Starting Page
585
Ending Page
Alternative Title
Abstract
Our study aims at enhancing Human-Robot Interaction, Collaboration, and Teaming (HRI/C/T) in industrial automation by developing a novel framework for real-time gesture control of a robotic hand. We use an Inertial Measurement Unit (IMU) sensor for precise orientation control of the end effector, and surface Electromyography (sEMG) sensors to detect muscle movements. The sEMG signals are processed by an Attention-based Multi Convolutional Neural Network (A-MCNN) for accurate gesture detection, enabling the robotic hand to mimic these gestures in real-time. Our method achieves notable results for gesture recognition, with the A-MCNN model attaining an accuracy of 97.89%, a precision of 97.49%, a recall of 97.71%, and an F1 score of 97.65%. This integration of IMU and sEMG technologies with advanced neural networks creates a responsive and intuitive control mechanism, improving safety, usability, and interaction of collaborative robots in shared workspaces. Our approach aims to transition towards Human-Robot Teaming (HRT), significantly advancing the seamless and safe integration of robots in industrial environments, enhancing productivity and collaboration.
Description
Keywords
Human-Robot Interaction and Collaboration, deep neural network, electromyography (emg) sensors, gesture recognition, hand gesture recognition, inertial measurement units (imus), multi-modal sensor fusion
Citation
Extent
10
Format
Geographic Location
Time Period
Related To
Proceedings of the 58th Hawaii International Conference on System Sciences
Related To (URI)
Table of Contents
Rights
Attribution-NonCommercial-NoDerivatives 4.0 International
Rights Holder
Local Contexts
Collections
Email libraryada-l@lists.hawaii.edu if you need this content in ADA-compliant format.