Aplicação de IA generativa em respostas automatizadas sobre interação humano-computador

Abstract

Generative Artificial Intelligence (GenAl), driven by Large Language Models (LLMs) such as GPT, has emerged as a promising tool for producing complex content. This study developed and evaluated a Retrieval-Augmented Generation (RAG) system using the Gemini model, based on scientific articles published in the Brazilian Symposium on Human Factors in Computing Systems (IHC-SBC) and the HCI International conference. The resulting model, named IAH- Chatbot, was trained on a corpus of 280 articles and assessed through both qualitative and quantitative approaches. Human evaluation involved six experts in Human-Computer Interaction (HCI), who rated the model highly for factual accuracy (mean 5.00), clarity (4.67), and relevance (4.50). The IAHChatbot showed competitive performance compared to established models such as ChatGPT, Claude, and Gemini, particularly in terms of conceptual and contextual alignment. However, it scored lower on completeness (3.67), likely due to limitations in the dataset, which lacked formal definitions and practical examples. Automated evaluation using the ragas toolkit showed varying performance depending on question type. The best results were observed in a question directly linked to a document in the training corpus (Context Recall: 1.0; Faithfulness: 1.0; Factual Correctness (F1): 0.56). Overall average scores were: Context Recall: 0.5; Faithfulness: 0.58; and F1: 0.77. The analysis indicates that the model often generated semantically correct responses even when the retrieved context was limited or not directly used. The findings suggest that the model holds strong potential as a pedagogical and research-support tool in the field of Human-Computer Interaction, especially when enhanced by curated textual data and iterative refinement. The proposed methodology is replicable and well-aligned with the needs of the Brazilian scientific context.