Vocabulário de testes instáveis entre linguagens de programação

Abstract

Context: Regression testing is a verification and validation activity of systems present in modern software engineering. In this activity, tests may fail without any implementation changes, characterizing them as unstable tests (flaky test). This type of instability can delay software release and reduce confidence in the tests. One way to identify such unstable tests is through test re-execution, but this comes with a high computational cost. An alternative to re-execution is static code analysis of the test cases, identifying patterns related to instability. In this context, so far, only vocabulary-based works focusing on single language applications (Java, Python, JavaScript) have been observed. To understand the limits and possibilities of this approach, it is interesting to check the intersection of unstable vocabularies across different languages. Objective: The objective of this work was to evaluate the vocabulary technique for predicting unstable tests in JavaScript, Java, and Python applications. Method: To achieve this objective, a dataset was constructed with unstable test cases present in open-source projects on GitHub that utilize JavaScript and the dataset from previous works for Python and Java was used. Classification models were then created, considering the instability vocabulary between different projects and languages. After evaluating the models’ accuracy, we examined the words with the highest information gain for predicting instability across languages. Results: We created the ShakerJS tool, which managed to identify 102 flaky tests in 36% of 36 relevant projects in JavaScript using CPU and memory stress. The terms with the highest information gain for predicting flaky tests in this set were: await, async, if, and path. In JavaScript, similarly to Java, the RF model had the best accuracy (0.96), while the DT model slightly surpassed RF in terms of recall (0.77 vs 0.63). The results are better when the model is trained and tested within the same project scope and language. Some projects achieve results of 100% accuracy and recall. We noted that the quality of the models concerning completely unknown intermittent failures is low. We observed the low quality of the models regarding intermittent failures across different programming languages: JavaScript, Java, and Python. The best result was for the model trained in JavaScript and tested in Java, which presented an F1 of 0.63. However, regarding the unstable vocabulary across languages, the results are positive. Between the programming languages Python and JavaScript, we can find terms related to the following root causes of flaky tests: asynchronous communication (await, 400, data), concurrency, graphical user interface events (plot, page, button, click, and width), time dependency (time), and resource leaks (source, data, args). Between the Java and JavaScript languages, we can verify the intersection of terms related to concurrency (manager), asynchronous waiting (await), and resource leaks (new, class). Between the Python and Java languages, we can find terms related to asynchronous communication (waitFor), concurrency (get, select), time dependency (date), and resource leaks (evaluate, catch). Conclusions: This work presents relevant results for a more efficient identification of unstable tests in projects that utilize JavaScript, Java, and Python. There is an intersection between contexts and root causes of instabilities from different projects and languages. However, using only the pure content of the source code may reduce the performance of prediction models in a scenario with different projects and languages.