Reconhecimento e personalização de gestos das mãos para interação com páginas web por meio de visão computacional e aprendizado de máquina

Abstract

This paper presents the Hand Mouse browser extension, developed to enable mouse control on web pages through the user’s hand gestures, using Computer Vision and Machine Learning techniques. This approach aims to enhance flexibility and ergonomics, offering a natural alternative to the traditional use of the mouse, potentially reducing the risk of repetitive strain injuries. The main innovation of this paper is the development of the gesture customization feature, allowing the user to define and train new gestures to perform actions such as moving the mouse cursor, clicking, scrolling up, scrolling down, going back a page, going forward a page, and free movement. To achieve this objective, a set of gesture images was created, which served as input for the TensorFlow.js Convolutional Neural Network (CNN) Hand Pose Detection model to extract hand key points (landmarks). These data were then used to test two classification models based on K-Nearest Neighbors (KNN) and Support Vector Machines (SVM) algorithms, selected for their application in related works and their tested results in gesture classification. During the classifier selection phase, tests were conducted to determine the minimum number of images needed to maximize the model’s accuracy, the influence of the distance between the hand and the camera, and the similarity between the gestures in the recognition process. The results show that the KNN model achieved satisfactory accuracy and F1 score values in gesture recognition, even under conditions with a reduced training set, variable distances, and gesture similarities. Within the Hand Mouse extension, a web page interface was developed that allows the user to register new gestures and retrain the KNN classifier. It is concluded that this paper not only achieved its goals of developing a functional tool to control the mouse using gestures but also paved the way for future research and improvements in the field of human-computer interaction, especially regarding the customization and usability of gesture-based interfaces.