HCAI/metabench

--- pretty_name: metabench configs: - config_name: ARC data_files: - split: primary path: "arc.parquet" - split: secondary path: "arc-secondary.parquet" - config_name: GSM8K data_files: - split: primary path: "gsm8k.parquet" - split: secondary path: "gsm8k-secondary.parquet" - config_name: HellaSwag data_files: - split: primary path: "hellaswag.parquet" - split: secondary path: "hellaswag-secondary.parquet" - config_name: MMLU data_files: - split: primary path: "mmlu.parquet" - split: secondary path: "mmlu-secondary.parquet" - config_name: TruthfulQA data_files: - split: primary path: "truthfulqa.parquet" - split: secondary path: "truthfulqa-secondary.parquet" - config_name: Winogrande data_files: - split: primary path: "winogrande.parquet" - split: secondary path: "winogrande-secondary.parquet" --- # Metabench **Disclaimer**: The metabench project is ongoing. Running and evaluating models on metabench is not yet fully supported, please proceed with caution. ## Dataset Description - **Repository:** [https://github.com/adkipnis/metabench](https://github.com/adkipnis/metabench) - **Paper:** [https://arxiv.org/abs/2407.12844](https://arxiv.org/abs/2407.12844) - **Point of Contact:** [[e-mail]]([e-mail]) ### Dataset Summary `metabench` consists of a subset of six common language benchmarks (ARC, GSM8K, HellaSwag, MMLU, TruthfulQA, Winogrande) which have been selected for being highly informative. The performance of models on the combined set of `metabench` testing items (n=858) has been demonstrated to predict the average performance of models on the full original benchmarks (n=28,632) within 1% error. We also provide a secondary `metabench` dataset which contains a strictly distinct set of items from the primary dataset (n=751). This dataset facilitates slightly less accurate prediction, but is nevertheless within 1% error. ### Supported Tasks and Leaderboards - `metabench_arc`: The dataset can be used to evaluate models on a subset of ARC items in a 25-shot regime. The standard benchmark metrics can be computed, as with the original ARC benchmark. Item Response Theory latent ability estimations can also be computed using the scripts [here](https://github.com/adkipnis/metabench). - `metabench_gsm8k`: The dataset can be used to evaluate models on a subset of GSM8K items in a 5-shot regime. The standard benchmark metrics can be computed, as with the original GSM8K benchmark. Item Response Theory latent ability estimations can also be computed using the scripts [here](https://github.com/adkipnis/metabench). - `metabench_hellaswag`: The dataset can be used to evaluate models on a subset of HellaSwag items in a 10-shot regime. The standard benchmark metrics can be computed, as with the original HellaSwag benchmark. Item Response Theory latent ability estimations can also be computed using the scripts [here](https://github.com/adkipnis/metabench). - `metabench_mmlu`: The dataset can be used to evaluate models on a subset of MMLU items in a 5-shot regime. The standard benchmark metrics can be computed, as with the original MMLU benchmark. Item Response Theory latent ability estimations can also be computed using the scripts [here](https://github.com/adkipnis/metabench). - `metabench_truthfulqa`: The dataset can be used to evaluate models on a subset of TruthfulQA items in a 0-shot regime. The standard benchmark metrics can be computed, as with the original TruthfulQA benchmark. Item Response Theory latent ability estimations can also be computed using the scripts [here](https://github.com/adkipnis/metabench). - `metabench_winogrande`: The dataset can be used to evaluate models on a subset of Winogrande items in a 5-shot regime. The standard benchmark metrics can be computed, as with the original Winogrande benchmark. Item Response Theory latent ability estimations can also be computed using the scripts [here](https://github.com/adkipnis/metabench). ### Languages All data are presented in Standard American English. See the relevant benchmark repositories for further details. ## Dataset Structure ### Data Instances Each instance is presented as in the original benchmark, along with a `metabench_idx` and the data for each shot for cases where a few-shot prompt is provided, plus the original benchmark fields for each shot in the few-shot cases. ### Data Fields See the relevant benchmark repositories for further details. ### Data Splits All instances ought to be used as testing instances only. For training and validation sets, please utilise the training and validation splits from the original benchmark repositories. ## Dataset Creation ### Curation Rationale This dataset has been created to offer a redundancy-free, minimal and general benchmark for the evaluation of language models. We foresee its use in prototyping and testing new models using a fraction of the resources that would be required for evaluation on the full standard benchmarks. ### Source Data The source datasets are: - **ARC-Challenge**: [https://huggingface.co/datasets/allenai/ai2_arc](https://huggingface.co/datasets/allenai/ai2_arc). - **GSM8K**: [https://huggingface.co/datasets/openai/gsm8k](https://huggingface.co/datasets/openai/gsm8k). - **HellaSwag**: [https://huggingface.co/datasets/Rowan/hellaswag](https://huggingface.co/datasets/Rowan/hellaswag). - **MMLU**: - Original version: [https://huggingface.co/datasets/cais/mmlu](https://huggingface.co/datasets/cais/mmlu) - Evaluation version: [https://huggingface.co/datasets/hails/mmlu_no_train](https://huggingface.co/datasets/hails/mmlu_no_train) - **TruthfulQA**: [https://huggingface.co/datasets/truthfulqa/truthful_qa](https://huggingface.co/datasets/truthfulqa/truthful_qa) - **Winogrande**: [https://huggingface.co/datasets/allenai/winogrande](https://huggingface.co/datasets/allenai/winogrande) #### Initial Data Collection and Normalization The filtering process is described in [Kipnis et al. (2024)](https://arxiv.org/abs/2407.12844). #### Who are the source language producers? See the relevant benchmark repositories for further details. ### Annotations See the relevant benchmark repositories for further details. #### Annotation process See the relevant benchmark repositories for further details. #### Who are the annotators? See the relevant benchmark repositories for further details. ### Personal and Sensitive Information See the relevant benchmark repositories for further details. ## Considerations for Using the Data ### Social Impact of Dataset Use of this dataset will encourage more economical and efficient evaluation of language models, contributing to the acceleration of progress in the field towards more capable systems. Any other impacts are shared by the source benchmarks used to create it, so please see those for further discussion. ### Discussion of Biases Instance selection in this benchmark has been biased by the population of models used to fit the psychometric models and any idiosyncracies therein. ### Other Known Limitations No relevant limitations noted. ## Additional Information ### Dataset Curators This dataset was curated by Alex Kipnis, Konstantinos Voudouris, Luca M. Schulze Buschoff, and Eric Schulz. It is based on datasets collected by others. See the relevant benchmarks for further details. ### Licensing Information [CC BY-NC-SA 4.0 License](https://creativecommons.org/licenses/by-nc-sa/4.0/). ### Citation Information ``` @article{metabench, author = {Alex Kipnis and Konstantinos Voudouris and Luca M. Schulze Buschoff and Eric Schulz}, title = {metabench - A Sparse Benchmark to Measure General Ability in Large Language Models}, journal = {arXiv preprint arXiv:2407.12844}, year = {2024}, } ``` ### Contributions Thanks to [@kvoudouris](https://huggingface.co/kvoudouris) for adding this dataset.pretty_name: 元基准（metabench） configs: - config_name: ARC data_files: - split: 主集 path: "arc.parquet" - split: 副集 path: "arc-secondary.parquet" - config_name: GSM8K data_files: - split: 主集 path: "gsm8k.parquet" - split: 副集 path: "gsm8k-secondary.parquet" - config_name: HellaSwag data_files: - split: 主集 path: "hellaswag.parquet" - split: 副集 path: "hellaswag-secondary.parquet" - config_name: MMLU data_files: - split: 主集 path: "mmlu.parquet" - split: 副集 path: "mmlu-secondary.parquet" - config_name: TruthfulQA data_files: - split: 主集 path: "truthfulqa.parquet" - split: 副集 path: "truthfulqa-secondary.parquet" - config_name: Winogrande data_files: - split: 主集 path: "winogrande.parquet" - split: 副集 path: "winogrande-secondary.parquet" # 元基准（metabench） **免责声明**：本元基准项目仍在开发中。目前尚未完全支持在元基准上运行与评估模型，请谨慎操作。 ## 数据集说明 - **代码仓库**：[https://github.com/adkipnis/metabench](https://github.com/adkipnis/metabench) - **相关论文**：[https://arxiv.org/abs/2407.12844](https://arxiv.org/abs/2407.12844) - **联系方式**：[[e-mail]]([e-mail]) ### 数据集概览 `metabench` 由六个常用语言基准数据集的子集构成（ARC、GSM8K、HellaSwag、MMLU、TruthfulQA、Winogrande），这些子集因具备高度信息性而被选中。实验表明，模型在 `metabench` 测试集（共858条样本）上的表现，可以在1%的误差范围内预测模型在完整原始基准数据集（共28632条样本）上的平均性能。我们还提供了副元基准数据集，其样本集与主数据集严格互斥（共751条样本）。该副集的预测精度略低，但仍能控制在1%的误差范围内。 ### 支持任务与排行榜 - `metabench_arc`: 该数据集可用于在25-shot少样本（Few-shot）设置下评估模型在ARC子集上的性能。可按照原始ARC基准的方式计算标准基准指标。此外，可通过[此处](https://github.com/adkipnis/metabench)提供的脚本计算项目反应理论（Item Response Theory, IRT）潜在能力估计值。 - `metabench_gsm8k`: 该数据集可用于在5-shot少样本（Few-shot）设置下评估模型在GSM8K子集上的性能。可按照原始GSM8K基准的方式计算标准基准指标。此外，可通过上述代码仓库的脚本计算项目反应理论潜在能力估计值。 - `metabench_hellaswag`: 该数据集可用于在10-shot少样本（Few-shot）设置下评估模型在HellaSwag子集上的性能。可按照原始HellaSwag基准的方式计算标准基准指标。此外，可通过上述代码仓库的脚本计算项目反应理论潜在能力估计值。 - `metabench_mmlu`: 该数据集可用于在5-shot少样本（Few-shot）设置下评估模型在MMLU子集上的性能。可按照原始MMLU基准的方式计算标准基准指标。此外，可通过上述代码仓库的脚本计算项目反应理论潜在能力估计值。 - `metabench_truthfulqa`: 该数据集可用于在0-shot零样本（Zero-shot）设置下评估模型在TruthfulQA子集上的性能。可按照原始TruthfulQA基准的方式计算标准基准指标。此外，可通过上述代码仓库的脚本计算项目反应理论潜在能力估计值。 - `metabench_winogrande`: 该数据集可用于在5-shot少样本（Few-shot）设置下评估模型在Winogrande子集上的性能。可按照原始Winogrande基准的方式计算标准基准指标。此外，可通过上述代码仓库的脚本计算项目反应理论潜在能力估计值。 ### 语言说明 所有数据均采用标准美式英语呈现。更多细节请参阅各原始基准数据集的代码仓库。 ## 数据集结构 ### 数据实例 每个实例均保留原始基准数据集的格式，同时包含`metabench_idx`字段；对于需要少样本提示的场景，还会提供各样本的提示数据，以及少样本场景下各样本对应的原始基准字段。 ### 数据字段 更多细节请参阅各原始基准数据集的代码仓库。 ### 数据划分 所有实例仅应用作测试样本。如需训练集与验证集，请使用各原始基准数据集对应的训练与验证划分。 ## 数据集构建 ### 筛选依据 本数据集旨在为大语言模型（Large Language Model, LLM）评估提供一个无冗余、极简且通用的基准。我们预期其可用于原型开发与新模型测试，所需资源仅为在完整标准基准上进行评估的一小部分。 ### 源数据集 源数据集包括： - **ARC-Challenge**：[https://huggingface.co/datasets/allenai/ai2_arc](https://huggingface.co/datasets/allenai/ai2_arc) - **GSM8K**：[https://huggingface.co/datasets/openai/gsm8k](https://huggingface.co/datasets/openai/gsm8k) - **HellaSwag**：[https://huggingface.co/datasets/Rowan/hellaswag](https://huggingface.co/datasets/Rowan/hellaswag) - **MMLU**： - 原始版本：[https://huggingface.co/datasets/cais/mmlu](https://huggingface.co/datasets/cais/mmlu) - 评估版本：[https://huggingface.co/datasets/hails/mmlu_no_train](https://huggingface.co/datasets/hails/mmlu_no_train) - **TruthfulQA**：[https://huggingface.co/datasets/truthfulqa/truthful_qa](https://huggingface.co/datasets/truthfulqa/truthful_qa) - **Winogrande**：[https://huggingface.co/datasets/allenai/winogrande](https://huggingface.co/datasets/allenai/winogrande) #### 初始数据收集与标准化 筛选过程详见[Kipnis等人（2024）](https://arxiv.org/abs/2407.12844)。 #### 源文本创作者 更多细节请参阅各原始基准数据集的代码仓库。 ### 标注信息 更多细节请参阅各原始基准数据集的代码仓库。 #### 标注流程 更多细节请参阅各原始基准数据集的代码仓库。 #### 标注人员 更多细节请参阅各原始基准数据集的代码仓库。 ### 个人与敏感信息 更多细节请参阅各原始基准数据集的代码仓库。 ## 数据使用注意事项 ### 数据集的社会影响 使用本数据集将推动大语言模型评估变得更加经济高效，有助于加速本领域向更先进的系统迈进。本数据集的其他影响与用于构建它的源基准数据集一致，详细讨论请参阅源基准的相关文档。 ### 偏差说明 本基准的实例选择受到用于拟合心理测量模型的模型群体及其特性的影响，因此存在一定偏差。 ### 其他已知局限性 暂无相关局限性说明。 ## 补充信息 ### 数据集整理者 本数据集由Alex Kipnis、Konstantinos Voudouris、Luca M. Schulze Buschoff与Eric Schulz整理，其基于他人收集的数据集构建。更多细节请参阅各原始基准数据集的相关文档。 ### 许可协议 [CC BY-NC-SA 4.0许可协议](https://creativecommons.org/licenses/by-nc-sa/4.0/)。 ### 引用信息 @article{metabench, author = {Alex Kipnis and Konstantinos Voudouris and Luca M. Schulze Buschoff and Eric Schulz}, title = {metabench - A Sparse Benchmark to Measure General Ability in Large Language Models}, journal = {arXiv preprint arXiv:2407.12844}, year = {2024}, } ### 贡献致谢 感谢[@kvoudouris](https://huggingface.co/kvoudouris)为本数据集添加至平台的工作。