Reconhecimento e personalização de gestos para interação com sistema operacional por meio de visão computacional e aprendizado de máquina

Abstract

This work proposes the development of an application for interaction with the Windows 11 operating system through customized hand gestures, using Computer Vision and Machine Learning. The objective is to promote accessibility, offering an alternative to the use of traditional input devices. The methodology utilizes the MediaPipe framework for the extraction and mathematical normalization of spatial hand coordinates from webcam image captures. For the training and evaluation of the models, a dataset containing samples of the customized gestures was built, in addition to the creation of a test dataset composed of images captured from a different user under varying lighting conditions, aiming to validate the system’s generalization. In the recognition and classification stage, the K-Nearest Neighbors (KNN) and Support Vector Machines (SVM) algorithms were evaluated. To solve false positive and overfitting issues, an architecture with a generic Background Class, composed of American Sign Language (ASL) samples, was implemented, coupled with a Negative Mining algorithm to avoid geometric label conflicts. The results demonstrate that the KNN algorithm configured with three neighbors (𝐾 = 3) was the chosen option for continuous classification due to its geometric transparency and suitability for the rejection architecture, achieving stability with a reduced number of images per gesture, registering 97.87% accuracy and a 96.93% Macro F1-Score. Additionally, the system shows potential to provide flexibility and autonomy through an intuitive graphical interface, which simplifies the management of the custom database and the mapping of functional gestures to computer actions, without requiring prior knowledge in artificial intelligence.