Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimization - Artifacts

Trajectories of Hyperparameter Optimization (HPO) runs as presented in the paper Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimization.Code for the analysis of the four datasets (.csv files) can be found in the accompanying GitHub repository: https://github.com/slds-lmu/paper_2024_reshufflingData contains the following columns where the validation and test performance of the incumbent are tracked over time in the form of trajectoriesiteration (iteration of an HPO run)orig_iteration (original iteration id of the incumbent at the current iteration)valid (validation performance)test (test performance)overtuningseed (replication id)classifier (learning algorithm)data_id (data set id)train_valid_size (size of the set used for training and validation)resampling (resampling method)metric (performance metric)method (post selection method and resampling method)optimizer (HPO algorithm)