Uso de large language models (LLMs) com retrieval augmented generation (RAG) e web scraping no processamento de linguagem natural

Abstract

This work develops and evaluates the implementation of pipelines using RAG architecture to answer questions about an academic institution. The main objective was to build pipelines capable of processing information from different document formats — JSON, PDF, and HTML — collected automatically and with minimal possible manipulation through different approaches and tools for data scraping from the institution’s portals. This process involved the construction of datasets with the extracted data, the processing and vectorization of this data with different embedding models, and the integration with different RAG strategies that were considered according to the scope of the data. The LLMs Gemma-2b and Llama3.2-1b were used to generate text based on the context retrieved by the pipelines. The results were evaluated with the RAGAS framework and demonstrated that the synergy between all stages of the system is essential, involving the quality of the data source, the precision of the information retriever, and the fdelity of the LLM model. While the pipeline for structured data (JSON) achieved good performance with the Gemma-2b model, the growing complexity of the PDF and HTML fles pipelines exposed the limitations of the implemented retrieval trategies, which proved insuffcient to handle semantic noise, resulting in a signifcant drop in performance. It is concluded that the viability of chatbots with smaller-scale LLMs depends on increasingly robust RAG strategies, capable of providing a clean and precise context to compensate the models’ limitations.