Using design of experiments during the process of tuning hyperparameters in machine learning algorithms

Hyperparameter tuning is a very important process to maximize the performance of any machine learning model. Currently, there is still no general agreement on how to fine tune hyperparameter. In this study, Design of Experiment (DOE) is proposed as a methodology for the process of hyperparameters tuning in machine learning algorithm. A dataset from lime-sorting computer vision model using Convolutional Neural Network (CNN) are used in this study. Five hyperparameters of the CNN model were focused on this research, including Batch size, Epoch, Learning rate, Decaying rate, and Momentum. The performance of the CNN model is observed from the weighted average validate F1-score which is used as a response variable in our experimental design. Sequential experimentation is used in hyperparameter tuning process. A factorial design was performed to screen out unimportant factors. The fitted model is then used to determine a path of steepest ascent. Then, a Central Composite Design (CCD) was performed to determine the best settings of the hyperparameters of every model. A methodology and discussion are detailed throughout the paper. Using design of experiments to tune hyperparameters not only helps screening out the unimportant factors in the initial step of experimentation, but it also considered the importance of the main effects and their interactions, produce fewer training runs, and help decrease the complexity of machine learning model.