Geração de pipelines de gerenciamento de dados escaláveis para arquiteturas de inteligência artificial

Abstract

This study proposes the development of a scalable data management pipeline for the retail sector, focusing on automation and optimization of data flow in cloud computing environments. The main motivation was the growing demand for solutions capable of integrating large volumes of transactional data, enabling analysis and decision-making support. To address this, an architecture was designed and implemented using orchestration tools such as Dagster and distributed processing libraries like Dask, within an environment based on the Google Cloud Platform (GCP). The pipeline evaluation considered performance metrics such as execution time, CPU and memory usage, in addition to operational costs, comparing two cluster configurations (c2-standard-4 and c2-standard-8). The results demonstrated that the pipeline is efficient and scalable, with the lower-capacity configuration offering the best cost-effectiveness, although the intensive data granularity increases the costs of read and write operations. This work contributes a modular and adaptable solution, applicable to various analytical scenarios, promoting operational efficiency and supporting artificial intelligence in the retail domain