Transferência de timbre utilizando auto encoder variacional e rede adversarial generativa

Abstract

Technological progress in neural architectures has revolutionized the field of audio, pushing the boundaries of creativity and making audio technologies more accessible. These advancements enable the creation of new sound experiences, facilitate music production, and promote inclusivity, benefiting both professionals and amateurs in the world of music and audio. This work proposes the development of a timbre transfer model using Variational Autoencoders (VAE) and Generative Adversarial Networks (GAN). The timbre transfer technique aims to convert the sound of one musical instrument só that it sounds as if it were played by another, preserving musical characteristics such as melody and dynamics. The model combines the ability of VAEs to learn compact and robust latent representations with the ability of GANs to generate realistic, high-quality outputs. The pre-processing process includes transforming the audio into Mel spectrograms, removing silences, and normalizing. The model training is organized in cycles, using multiple loss functions to ensure the quality of the outputs. Inference involves reconstructing the audio using the Griffin-Lim algorithm and evaluating the quality of the outputs generated through objective and subjective metrics. Visualization techniques, such as t-SNE, are used to analyze the latent representations and separation of timbre clusters. This study aims to contribute to the advancement of sound manipulation using artificial intelligence, promoting creative and accessible solutions for music production and other applications related to timbre transfer. The research was guided by studies of timbre, musical perception and cognition, signal and audio processing, as well as tests of existing models. After all the study and development, it was possible to find the model that best fits the project proposal, obtaining satisfactory results that present a reduction of up to 53% in losses and an increase of up to 237% in clustering metrics, ensuring that the generator and the discriminator improve their respective tasks in a balanced way and indicating a significant improvement in the performance of the models throughout the training.