Comparação entre os modelos pré-treinados GPT-3 e BERT na estimativa de esforço de software por analogia a partir de requisitos textuais

Abstract

Estimating the software effort required to develop each of user requirements is one of the most important steps of the software development cicle. There are several techniques to perform this activity, but Artificial Intelligence, such as Machine Learning methods associated with Natural Language Processing (NLP), can contribute by automating part of this estimation process, based on requirements texts (e.g. user stories). Analogy-based software effort estimation (ASEE) requires knowledge about historical project data and is rarely used during the planning phase of the development process. Still, normally in the initial phase, it is common to have requirements in textual format. Therefore, this work aimed to perform a comparation of the text representation models GPT-3 and BERT when applied on the ASEE from the analysis of textual user requirements. For that, this work applied the contextualized pre-trained model GPT-3 in the representation of textual features for the inference of software effort estimates by analogy and then compared its performance with that of the SE3M – Software Effort Estimation Embedding Model –, which uses the pre-trained language model BERT. For the pre-processing of the requirements, regular expressions were used, and for the inference of the estimates, feedforward neural networks were trained. To evaluate the model, two experiments with different partitioning of the text corpus were performed, from which the metrics Mean Squared Error (MSE), Mean Absolute Error (MAE) and Median Absolute Error (MdAE) were extracted. Once the metrics of the two experiments were obtained, they were compared with the results of the SE3M model. In the end, a better performance was observed from the model that used the GPT-3 compared to the one that used the BERT model (SE3M). Of the metrics obtained, the MAE of 3.80±1.20 (95% confidence interval) stands out, which comes from the experiment that used a cross-repository partitioning. This result is similar to that obtained by the SE3M and Deep-SE (also compared) models, but with the advantage that the GPT-3 model was used without fine-tuning. The main limitation raised in using GPT-3 was the cost, which is required both for extracting the text representation and for fine-tuning.