Aplicação de knowledge distillation em transfer learning

Abstract

The development of deeper neural network models has been made possible by advancements in hardware, aiming to achieve the best predictive performance through the construction of increasingly deeper models. However, it is necessary to be aware of the impacts and constraints that large models bring. Some implications include the requirement for computationally powerful hardware, long training times, computational effort to perform inference with these models, and even the environmental impact that these processes can cause through high energy consumption. One way to reduce energy consumption during the training process is through the use of Transfer Learning. However, inference still needs to be performed using the computationally expensive model. This work aims to use the Knowledge Distillation compression method to construct a student network with lower computational cost, approximating the extracted characteristics when performing Transfer Learning with the teacher network. The purpose is to find student models that can replace teacher networks when performing model inference. To achieve this goal, various architectures and teacher models were evaluated, comparing how well student models approximate the characteristics and predictive performance of the teacher model across multiple tasks. To approximate the characteristics of teacher models, three student architectures were employed. The results obtained through these student networks are comparable or better than the results achieved through the VGG-16, ResNet50V2, InceptionV3, and DenseNet201 teacher networks on 2 out of 3 datasets (MNIST and Fashion-MNlST). Using the SVM for classification of the extracted and approximated characteristics, the best F1-scores in the MNIST and Fashion-MNlST datasets were 0.98 +- 0.001 and 0.96 +- 0.004, respectively. In the CIFAR-10 dataset, the best result was obtained by the DenseNet201 teacher network, with an F1-score of 0.71 +- 0.005. Due to the results achieved using student networks much less complex than teacher networks, it is possible to suggest that the performances obtained indicate the possibility of replacing the teacher model with the student model in the training and inference process.