Detecção de outliers em pipelines de dados

Abstract

The technological advances of the last decades allowed a huge increase in the rate and amount of data generated by computing systems. Data flow management has become relevant for obtaining information. Data pipelines are used to transport the data from its source to different sorts of targets, transforming the data itself whether necessary. Data pipelines are sensitive to changes that occur both in the extraction process and in their business rules, leading to data inconsistency. This study proposes a solution to identify behavior changes in data pipelines (outliers), through the use of machine learning. Some machine learning algorithms have been applied during the data loading process by using the pipeline metrics to identify anomalies. The proposed work has been evaluated throughout some case studies that use a publicly available dataset and another created in the context of the study, namely, US Accidents Updated until Dec 2020 and Local Machine Metrics. An environment was created using Docker. It is comprised of three containers: one using StreamSets for creating data pipelines; one using MySQL database to store pipeline metrics, and one using Jupyter to perform exploratory analysis of the data and testing of the algorithms. The algorithms used in this study were PCA and Dbscan. The obtained results are considered satisfactory since the pipelines that had divergent comportment were correctly identified and reported.