Tesutã: um guia para apoiar a condução de testes metamórficos em chatbots

Abstract

Context: With the increasing use of conversational agents (chatbots), users can interact with machines through natural language. This rise highlights the need for a guide to support testing activities in this domain. Furthermore, metamorphic testing has emerged as an effective approach to address unique challenges in this area by offering automated ways to identify flaws in conversational agents. Problem: The growing demand for chatbots has created complex challenges in designing, implementing, and testing these conversational agents. Objective: To propose a guide that provides clear and practical guidelines for testing chatbots, helping professionals select the best tools and methods for their specific needs. Method: A systematic mapping study was conducted to identify the state of the art in chatbot testing. Subsequently, specific chatbot characteristics that differentiate their testing from traditional systems were analyzed. Based on this analysis, a set of guidelines was defined to support chatbot testing. To validate the effectiveness of these guidelines, an evaluation with students was conducted to analyze the completeness and correctness of the generated tests. Metamorphic testing was identified as a relevant technique during the systematic mapping due to its ability to validate test cases without explicit expected results. This technique was incorporated into the guide, Tesuta, as a specific guideline to address the challenges of chatbot testing, especially in natural language processing scenarios. Results: This work developed the Tesuta guide, which provides an overview of tools and methods for testing chatbots. The evaluation of the guide, through experiments with students, revealed that it facilitates the selection of suitable approaches, increasing testing effectiveness. The results showed that Tesuta contributes to fault detection and improves chatbot quality, highlighting its potential as a valuable tool in an ever-evolving domain. Final Remarks: The evaluation of the Tesuta guide demonstrated its potential in applying metamorphic testing to chatbots. While Tesuta achieved a significant completeness rate of 75.70% for the identified requirements, the variability in participants’ performance indicated the need for adjustments to improve its accessibility.