Pipeline para centralização e visualização de logs em sistemas distribuídos

Abstract

The growing adoption of container-based architectures orchestrated by Kubernetes has increased the demand for efficient observability mechanisms capable of providing operational visibility and assisting in fault detection within distributed environments. In this context, this work proposes the implementation and analysis of a log-based observability pipeline that performs collection, enrichment, storage, and visualization of log data using Fluent~Bit, Elasticsearch, and Kibana deployed in a Kubernetes cluster provisioned with Minikube. The motivation for this study lies in the increasing importance of structured monitoring in cloud-native systems, where the volume of logs is high and manual inspection becomes impractical. The main objective was to develop a functional architecture that would allow a practical understanding of the complete log-processing workflow in Kubernetes. The methodology included deploying an experimental environment, configuring a synthetic log generator, setting up Fluent~Bit as a collection agent, installing the Elastic Stack, and conducting experiments to evaluate the pipeline’s performance. The results showed that the solution was capable of collecting and indexing more than 150,000 log entries without losses, maintaining temporal consistency and enabling advanced queries and rich visualizations through Kibana. The automatic enrichment of logs with metadata from the Kubernetes API significantly improved event contextualization and facilitated analysis. In conclusion, the proposed architecture successfully met the objectives, proving to be a viable option for educational use and prototyping, though limited by the single-node environment and synthetic data. Nonetheless, the study provides a foundation for future enhancements involving scalability, additional metrics, and integration with alerting systems.