Uma base textual em português destinada a pesquisas de processamento de linguagem natural aplicados a engenharia do software

Abstract

Applications of Natural Language Processing (NLP) mostly require large volumes of data. This is necessary in order to develop a robust application, typically based on machine learning algorithms. Much of the research in this field relies on specific and curated databases created by the authors themselves, which rarely reach the volume required by machine learning aplications. With the aim of providing a textual database in Brazilian Portuguese for the Software Engineering field, this work presents the process of automatic data retrieval and processing, extracted from questions and answers on the popular portal Stack Overflow, widely used by members of the field. Therefore, this work encompasses concepts of web crawling and web scraping, tools used to extract data from the internet, as well as preprocessing methods for the extracted texts using NLP techniques. Preprocessing is important because this type of data often contains HTML language characteristics and a wide range of irregular data, but the goal is to maintain the originality and coherence of the sentences, allowing future research to consistently utilize this database for various NLP tasks. The results present the approach used, its characteristics, and the challenges encountered. Finally, the generated database is presented, highlighting its main features.