Sistema de automatização de animação Lip Sync por algoritmo de alinhamento forçado

Abstract

This paper presents the development of a plug-in program for the Autodesk MAYA animation editor, with the aim of automating 3D lip sync animations in English and Japanese. The automation was performed using machine learning methods applied to phoneme recognition by forced alignment and speech emotion recognition by neural networks, using as input audio files, their transcriptions and sets of predefined viseme animations. For the acoustic models in English, pre-trained models made available by the Montreal Forced Alignment (MFA) API were used. However, due to alignment problems in the existing pre-trained model for Japanese, it was necessary to train a new model, adapted to the kana syllabic system. This approach simplified the Japanese phonetic dictionary, mitigating errors caused by the flexibility and wide lexical variety of the language. The tool developed aims to reduce the workload of animators, given that manually creating this type of animation is often considered repetitive and time-consuming. In addition, the proposed method improves the quality of animations by giving greater naturalness to lip movements, in contrast to more simplified approaches that only use time-domain analysis of sound waves. The technology presented has applications in several areas, such as cartoon animation, electronic games, computer graphics and any media that requires synchronized lip animations from audio files.