Improving model performance: comparing complete fine-tuning with parameter efficient language model tuning on a small, portuguese, domain-specific, dataset

Abstract

Deep neural networks, a class of machine learning algorithms, added a huge leap in performance for many different tasks since they won ImageNet competition in 2012. Among the benefited fields, Natural Language Processing (NLP) was specially impacted by the publication of “Attention is All you Need” paper, in 2017, which gave foundation to many posterior advancesin the field. Since then, models are getting progressively more accurate, at the cost of getting bigger and more expensive to train. Transfer Learning contributes by enabling the reuse of large Language Models pre-trained parameters, as they are expensive to optimize. It is possible tofine-tune them from the pre-trained model checkpoint for downstream (derived) tasks. This helps with computational costs of training such large models as well as it avoids the need to gather all data needed for such endeavour. Parameter Efficient Language-model Tuning (PELT) strategies tries to deepen fine-tuning advantages by at least maintaining final model performance whilefine-tuning as few parameters as possible. This enables two things: even less computational costs and competitive performance on small data sets for fine-tuning. This work leveraged these advantages in order to obtain better model performance on a legal text classification dataset,built during this work. BitFit performance on small, domain-specific, real-world dataset wascompared with complete fine-tuning performance. Results have shown that BitFit fine-tuningis more resistant to fine-tuning data noise and, perhaps, solves the “Catastrophic Forgetting”problem. Also, BitFit outperformed complete fine-tuning on 3 out 5 dataset versions. Finally, themodel was presented to and amused the brazillian Federal Court of Audits (from Portuguese: TCU - Tribunal de Contas da União).