HPOBench
收藏资源简介：
HPOBench是一个包含7个现有和5个新基准系列的数据集，总计超过100个多保真度基准问题。该数据集允许以可重复的方式运行这些可扩展的多保真度HPO基准，并通过容器隔离和打包各个基准。它还提供了代理和表格基准，用于计算上可承受但统计上可靠的评估。
HPOBench is a dataset comprising 7 existing and 5 novel benchmark suites, totaling over 100 multi-fidelity benchmark problems. This dataset enables reproducible execution of these scalable multi-fidelity HPO benchmarks, with individual benchmarks isolated and packaged via containers. It also provides surrogate and tabular benchmarks for computationally affordable yet statistically reliable evaluations.
HPOBench 数据集概述
数据集描述
HPOBench 是一个用于提供（多保真度）超参数优化基准的库，重点关注可重复性。
数据集状态
数据集使用示例
评估随机配置
python from hpobench.container.benchmarks.nas.tabular_benchmarks import SliceLocalizationBenchmark b = SliceLocalizationBenchmark(rng=1) config = b.get_configuration_space(seed=1).sample_configuration() result_dict = b.objective_function(configuration=config, fidelity={"budget": 100}, rng=1)
查询无保真度
python from hpobench.container.benchmarks.nas.tabular_benchmarks import SliceLocalizationBenchmark b = SliceLocalizationBenchmark(rng=1) config = b.get_configuration_space(seed=1).sample_configuration() result_dict = b.objective_function(configuration=config, fidelity={"budget": 50}, rng=1) result_dict = b.objective_function(configuration=config, rng=1)
获取搜索空间和保真度空间信息
python from hpobench.container.benchmarks.nas.tabular_benchmarks import SliceLocalizationBenchmark b = SliceLocalizationBenchmark(task_id=167149, rng=1) cs = b.get_configuration_space(seed=1) fs = b.get_fidelity_space(seed=1) meta = b.get_meta_information()
安装指南
安装步骤
bash git clone https://github.com/automl/HPOBench.git cd HPOBench pip install .
依赖项
- Singularity (版本 3.6)：安装指南
- ConfigSpace
- scipy
- numpy
容器化基准
运行本地基准
python from hpobench.benchmarks.ml.xgboost_benchmark_old import XGBoostBenchmark b = XGBoostBenchmark(task_id=167149) config = b.get_configuration_space(seed=1).sample_configuration() result_dict = b.objective_function(configuration=config, fidelity={"n_estimators": 128, "dataset_fraction": 0.5}, rng=1)
本地构建容器
bash cd hpobench/container/recipes/ml sudo singularity build xgboost_benchmark Singularity.XGBoostBenchmark
使用本地容器
python from hpobench.container.benchmarks.ml.xgboost_benchmark import XGBoostBenchmark b = XGBoostBenchmark(task_id=167149, container_name="xgboost_benchmark", container_source=./) config = b.get_configuration_space(seed=1).sample_configuration() result_dict = b.objective_function(config, fidelity={"n_estimators": 128, "dataset_fraction": 0.5})
配置文件
- hpobenchrc 文件：存储在
$XDG_CONFIG_HOME或~/.config/hpobench - Unix 套接字：存储在
$TEMP_DIR或/tmp
数据存储位置
- HPOBench 数据：
$XDG_CONFIG_HOME$XDG_CACHE_HOME$XDG_DATA_HOME
- OpenML 数据：
~/.openml/ - Singularity 容器缓存：
singularity cache clean
引用
bibtex @inproceedings{ eggensperger2021hpobench, title={{HPOB}ench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for {HPO}}, author={Katharina Eggensperger and Philipp M{"u}ller and Neeratyoy Mallik and Matthias Feurer and Rene Sass and Aaron Klein and Noor Awad and Marius Lindauer and Frank Hutter}, booktitle={Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)}, year={2021}, url={https://openreview.net/forum?id=1k4rJYEwda-} }
- 1HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO · 2022年
