Processo de desenvolvimento de uma feature store a partir de dados de web scraping

Abstract

With the growth of the electronic games industry and the dependence on data for decision making, the work seeks to present a data transformation process until it is significantly safe and ready for analysis or to be acquired by an machine model learning . As an example, let’s use event data from Counter Strike championship matches. This process consists of an web scraping step, where it was necessary to develop an script in Python to standardize the process, and another data transformation step using the Spark processing engine, since the amount of extracted data was considerably large. The result is an feature store, which is a set of data about the statistics of professional players in championships with prizes above 100 thousand dollars.