Técnicas otimizadas de mineração de dados com Hadoop

Abstract

Nowadays it is indisputable that the data is increasing exponentially and comes from the most diverse sources, this expansion leads to the creation of a mass of complex data. This mass of data is called Big Data and, linked to this concept, there is a need to create a data architecture that supports the extraction of information in a large and variable volume of data and even that makes the transformation of this data into information relevant to the more variable segments, quickly. This new reality requires computing new ways to keep up with the complexity of the data, among which is parallel and distributed computing, which uses a cluster of computers. However, this form of computation requires a specific knowledge from the user and in this context the Apache Hadoop appeared in order to solve the problems of distributed computing, through the application of a single open source framework seeking to isolate the programmer who works with large amounts of data from the need to address the traditional problems of distributed computing. The objective of this work is to apply data mining techniques in cluster together with partitioning techniques in the Hadoop framework and to verify the accuracy of the data and execution time obtained with this technique in comparison to the use of a single computer, in an attempt to demonstrate the effectiveness of Hadoop in storing and processing this data, which is in the proportion of petabytes daily, and its ability to reduce storage system costs and increase processing capacity.