Development of upper limb gestures recognition model for hearing and speech impaired patients using convoluted neural network

Upper limb gesture recognition plays a major role in overcoming many difficulties and inconveniences in human life, especially for individuals with speech disabilities and hearing impairment. The ability of machines to understand human activities and process their meaning can be utilized in a vast array of applications. One of the most specific applications is sign language analysis, prediction and recognition, which will aid effective communication between healthcare providers and disabled patients. This study provides a thorough state-of-the-art technique for developing upper limb gestures and sign language recognition models, predominantly based on a computational method called Deep Learning (DL). This study implements a Convolutional Neural Network (CNN), among other DL techniques, to develop a system using different stages such as data acquisition and generation, preprocessing, classification, and model building. The model was built using different stages to analyze, detect and recognize the generated (upper limb gestures data). Performance evaluation was carried out using accuracy, precision, loss function, RMSE, MSE and MAE metrics. The CNN model evaluation results are ۰.۲۹۱۳ Loss function, ۷۸.۷۳% Accuracy, ۰.۸۴۰۹ Precision, ۰.۷۳۹۶ Recall, ۰.۱۱۲۷ RMSE, ۰.۰۱۲۷ MSE and ۰.۰۲۴۵ MAE. The study, therefore, concludes that CNN can be applied to build a gesture recognition model based on its performance. Also, insights are provided in the field of gestures and sign language recognition to facilitate future research efforts and recommend the application of reinforcement learning for developing an automated embedded system.

Keywords:

CNN , Deep Learning , Gesture recognition , Machine Learning , Sign language , Speech disability , Upper Limb Gesture