Identificação de sinais em libras utilizando redes neurais: long short term memory

Abstract

The following document takes the Brazilian sign language (LIBRAS) as its main object of study. LIBRAS, despite being widely popularized today, has a relatively recent historical context and is currently still a paradigm for a large part of the Brazilian population. Using the advent of artificial intelligence, it is proposed the application of a recurrent neural network (RNN) of the Long-Short Term Memory (LSTM) type capable of identifying signals coming from an interlocutor through a simple camera and translating them. A database from the UFPE repository is taken as the starting point of the work, with 1364 expressions of LIBRAS and for each of the expressions, there are three (3) records in videos, totaling a database with 4089 records. Due to the current repository of the database having the domain partially corrupted, the records had to be obtained manually, in order to capture the records, an automated process by robot (RPA) was used. A total of 9.5% of the data were used, representing 130 expressions and 390 total records for work. During data augmentation, each video was subjected to format-specific filters, so that the records had a broader use of the database and in order to avoid concepts of overfitting. After obtaining the records and increasing the data used, the records were treated in such a way as to become numerical data, for this purpose the vídeos were read and the points of the body were mapped (using a CNN) during a specific interval of frames of the video, the interval for all videos was defined as 60 frames, in other words, for each of the 60 frames, the points of the body were captured and transformed into numerical sets to be subsequently submitted to LSTM. After mapping, the LSTM network was created and trained with the following arrangement of records: 70% of records for training (6461); 15% of records for validation (1384) and 15% for testing (1385); the LSTM network was able to obtain results in terms of precision, greater than 98%, errors smaller than 0.02%, considering the MAE; less than 0.06% considering MSE and less than 1% considering categorical cross-entropy. The practical tests were used with a cell phone camera in the form of a WebServer. In the tests, a limit was defined for the display of words on screen for real time, that is, if the network could not obtain an accuracy greater than 90% for the prediction of a given expression, the program would not be able to display any. Through the crude analysis of the practical tests, a precision of 47.2% of accuracy was confirmed. The work can add to the deaf-mute community and academic branches of this community.