BREEDS-Benchmarks

BREEDS-Benchmarks是由麻省理工学院MadryLab创建的一套大型子群体转移基准数据集，旨在评估模型对训练过程中未观察到的新数据子群体的泛化能力。该数据集利用现有数据集的类别结构，控制训练和测试分布中的数据子群体，从而合成现实分布转移。通过应用这种方法到ImageNet数据集，创建了一系列不同粒度的子群体转移基准。这些基准用于测量标准模型架构的敏感性以及现成训练时间鲁棒性干预措施的有效性。此外，通过人类基线验证了相应的转移是可处理的，确保了数据集在研究模型鲁棒性到分布转移方面的实用性。

BREEDS-Benchmarks is a large-scale subpopulation shift benchmark dataset developed by the MIT MadryLab, designed to evaluate a model's generalization capability to unseen novel data subpopulations that were not observed during training. This dataset leverages the categorical structure of existing datasets, regulates data subpopulations across both training and test distributions to synthesize realistic distribution shifts. By applying this methodology to the ImageNet dataset, a suite of subpopulation shift benchmarks with varying granularities is created. These benchmarks are used to measure the sensitivity of standard model architectures and the effectiveness of off-the-shelf training-time robustness interventions. Additionally, the corresponding distribution shifts are validated to be tractable via human baselines, thereby ensuring the dataset's utility in researching model robustness against distribution shifts.