Overtuning in Hyperparameter Optimization - Artifacts

Trajectories of Hyperparameter Optimization (HPO) runs as analyzed in the paper Overtuning in Hyperparameter Optimization.Code for the analysis of the datasets (.csv files) can be found in the accompanying GitHub repository: https://github.com/slds-lmu/paper_2025_overtuning.`overtuning_csvs.zip` contains already postprocessed HPO trajectories of different HPO runs with columns being self-explanatory.`reshuffling_raw_csvs.zip` contains raw HPO trajectories as produced in the experiments of the paper Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimization.In contrast to https://doi.org/10.6084/m9.figshare.27627504.v1 these are the raw HPO trajectories and not the trajectories over the incumbents.Additionally, HPO runs of HEBO with early stopping are included.This data contains the following columns (and some additional ones not explained here which are self-explanatory) where the validation and test performance of each proposed hyperparameter configuration are tracked over time in the form of trajectoriesiteration (iteration of an HPO run)valid (validation performance)test_retrained (test performance after retraining)seed (replication id)classifier (learning algorithm)data_id (data set id)train_valid_size (size of the set used for training and validation)resampling (resampling method)metric (performance metric)method (post selection method and resampling method)optimizer (HPO algorithm)