Metodologia baseada em visão computacional e aprendizado de máquina para guiar o projeto de sistemas de comunicação aumentativa e alternativa usando interação gestual personalizada

Abstract

Verbal communication is essential for socialization, building and sharing knowledge in society. People with speech impairment may experience difficulties in interaction and learning, family life, among other situations that can lead to emotional, social, and cognitive problems. Often, these people also have associated motor impairment, which further hinders their ability to interact with other people and the environment in which they are inserted. Augmentative and Alternative Communication (AAC) is a research area that involves the use of non-oral modes as a complement or substitute for spoken language, increasing the communicative interaction of individuals without orality. The AAC supported by Computer Vision (CV) systems can benefit from the recognition of the user’s remaining functional movements as an alternative design approach to interaction. In this thesis, a new methodology, named MyPGI (Methodology to yield Personalized Gestural Interaction), is presented, built to guide the design of AAC systems for people with motor and speech difficulties, using CV techniques and Machine Learning to enable personalized and non-invasive gestural interaction. Many of the works in the literature focus on specific situations and characteristics, offering little or no flexibility for people with disabilities and their different contexts of use, thus requiring people to adapt to the system instead of adjusting the system to different user needs. The MyPGI methodology was used in the development of an AAC system, named PGCA (Personal Gesture Communication Assistant), employing a low-cost approach, in experiments conducted with volunteers, including students with motor and speech difficulties. The results obtained with the experiments and also with interviews and usability evaluation conducted with professionals from the Special Education area indicate the feasibility of the proposed methodology and the system developed, in addition to adding knowledge about the specific challenges and needs of the target audience. The main scientific contribution of this thesis is, therefore, the definition of the MyPGI methodology, to promote greater autonomy for users of computer systems that have motor and speech difficulties and for their caregivers. The technological contribution of this thesis refers to the production of an AAC system as a way of applying the proposed methodology.