Brazilian sign language recognition through SEMG using deep learning time series classification models

Abstract

The hearing-impaired population relies mainly in sign language as a form of communication, which limits its integration since not everyone is a sign language speaker. To mitigate this problem, Sign Language Recognition (SLR) systems present an alternative to bridge the gap between spoken and sign languages. The SLR systems are presented in multiple modalities, as for example the Surface Electromyography (sEMG) processing using wearable devices presented in this work. This approach challenges various characteristics of the Surface Electromyography (sEMG) signal and its stochastic nature, with even more limitations when applied to dynamic movements and signs. With these factors in mind, this work leverages the nowadays ever growing area of Deep Learning and compares the performance of 10 state-of-the-art Time Series Classification models: Residual Networks (ResNet), eXplainable Convolutional Neural Network for Multivariate Time Series Classification (XCM),Multivariate Wavelet Decomposition Network (mWDN), Res-CNN, Multivariate Long Short Term Memory- Fully Convolutional Network (MLSTM-FCN), Time Series Transformer (TsiT), Omni-scale, Xception Time, Gated Multilayer Perceptron (gMLP) and Time Series Perceiver (TSPerceiver); on the task of processing the raw sEMG data acquired from the right forearm with a Myo𝑇𝑀 Armband without using the processing step of feature extraction. The database used contains 50 different words from the Brazilian Sign Language (Libras) executed by 11 volunteers, one of then deaf and fluent in this language. Three different scenarios of model training are presented and analyzed through six different metrics. The two first scenarios expect that the data of a new user is collected for training. The first scenario trains an individual model for each new user, using only the dataset collect from this user, while the second scenario adds the new user data to a larger dataset. The third scenario trains the model over data collected from other users without the necessity for collecting the data from that new user. For the results obtained, the Xception Time model attained up to 0.988 accuracy and over 0.98 over all analyzed metrics in the first training scenario for the dataset of one subject. For the second scenario, the best performance recorded was also obtained by the Xception Time model, with an accuracy of 0.651 in its best case. In the third case, however, performances were reduced, and the best result was obtained by ResNet, with an accuracy of 0.214. The first scenario shows the best overall performance for all models and indicates that the best training regime for the models should be individualized for each new user. The model with the best average performance across scenarios was Xception Time, achieving superior results in the first two scenarios. The third scenario is the one that facilitates the applicability of the trained models tonew users, but it obtained the worst results, which may have occurred due to restrictions on the number of volunteers in the database used and should be revisited in future experiments.