Abordagem de desenvolvimento de uma pipeline de migração de dados utilizando inteligência artificial

Abstract

Large language models (LLMs) and artificial intelligence are increasingly serving as tools to assist users in various functions and enhance productivity. Given the significant growth in data availability and the need to explore it, data manipulation using data engineering techniques has become a latent topic. Therefore, the purpose of this work is to combine these two aspects to assess, through different criteria, the capability of artificial intelligence tools in generating code for data migration and transformation. The expectation is that this approach will help mitigate existing challenges in the field, such as the shortage of qualified professionals in the job market and the inherent complexity of these tasks, which, through the use of these techniques, can be simplified.