Enhanced Human-Robot Teaming Through Attention Multi Convolutional Neural Network-Based Multi-Modal Sensor Fusion for Hand Gesture Recognition and Orientation Control

Abstract

Our study aims at enhancing Human-Robot Interaction, Collaboration, and Teaming (HRI/C/T) in industrial automation by developing a novel framework for real-time gesture control of a robotic hand. We use an Inertial Measurement Unit (IMU) sensor for precise orientation control of the end effector, and surface Electromyography (sEMG) sensors to detect muscle movements. The sEMG signals are processed by an Attention-based Multi Convolutional Neural Network (A-MCNN) for accurate gesture detection, enabling the robotic hand to mimic these gestures in real-time. Our method achieves notable results for gesture recognition, with the A-MCNN model attaining an accuracy of 97.89%, a precision of 97.49%, a recall of 97.71%, and an F1 score of 97.65%. This integration of IMU and sEMG technologies with advanced neural networks creates a responsive and intuitive control mechanism, improving safety, usability, and interaction of collaborative robots in shared workspaces. Our approach aims to transition towards Human-Robot Teaming (HRT), significantly advancing the seamless and safe integration of robots in industrial environments, enhancing productivity and collaboration.