Otimização de hiperparâmetros em machine learning utilizando uma surrogate e algoritmos evolutivos

Abstract

This work presents a new approach for hyperparameter optimization in Machine Learning algorithms. The idea is to build a surrogate with quasirandom numbers generated by Sobol's algorithm and then use an evolutionary algorithm (DE or PSO in this case) to perform the optimization on it. It is also implemented a way of performing more than one optimization with the evolutionary algorithm in the same execution of the method without increasing the computational cost in relation to other optimization methods. The objective is then to verify if the use of these two strategies would make the proposed method less likely to be trapped in a local minimum and also make it more consistent when compared to others. The focus of this work was the optimization of LightGBM applied to binary classification, however it is possible to expand it to other areas by making the necessary adaptations. In addition to the proposed method, Random Search and Bayesian Optimization are also used to optimize LightGBM models trained on 3 datasets. The results obtained show that the proposed approach with PSO is the most consistent of the 3 methods, however Bayesian Optimization still performs better in general.