open-llm-leaderboard-old/details_FabbriSimo01__GPT_Large_Quantized

Name: open-llm-leaderboard-old/details_FabbriSimo01__GPT_Large_Quantized
Creator: open-llm-leaderboard-old
Published: 2023-09-23 20:31:23
License: 暂无描述

Hugging Face2023-09-23 更新2024-06-22 收录

下载链接：

https://hf-mirror.com/datasets/open-llm-leaderboard-old/details_FabbriSimo01__GPT_Large_Quantized

下载链接

链接失效反馈

官方服务：

资源简介：

--- pretty_name: Evaluation run of FabbriSimo01/GPT_Large_Quantized dataset_summary: "Dataset automatically created during the evaluation run of model\ \ [FabbriSimo01/GPT_Large_Quantized](https://huggingface.co/FabbriSimo01/GPT_Large_Quantized)\ \ on the [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).\n\ \nThe dataset is composed of 3 configuration, each one coresponding to one of the\ \ evaluated task.\n\nThe dataset has been created from 1 run(s). Each run can be\ \ found as a specific split in each configuration, the split being named using the\ \ timestamp of the run.The \"train\" split is always pointing to the latest results.\n\ \nAn additional configuration \"results\" store all the aggregated results of the\ \ run (and is used to compute and display the agregated metrics on the [Open LLM\ \ Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)).\n\ \nTo load the details from a run, you can for instance do the following:\n```python\n\ from datasets import load_dataset\ndata = load_dataset(\"open-llm-leaderboard/details_FabbriSimo01__GPT_Large_Quantized\"\ ,\n\t\"harness_winogrande_5\",\n\tsplit=\"train\")\n```\n\n## Latest results\n\n\ These are the [latest results from run 2023-09-23T20:31:12.168542](https://huggingface.co/datasets/open-llm-leaderboard/details_FabbriSimo01__GPT_Large_Quantized/blob/main/results_2023-09-23T20-31-12.168542.json)(note\ \ that their might be results for other tasks in the repos if successive evals didn't\ \ cover the same tasks. You find each in the results and the \"latest\" split for\ \ each eval):\n\n```python\n{\n \"all\": {\n \"em\": 0.0,\n \"\ em_stderr\": 0.0,\n \"f1\": 3.3557046979865775e-05,\n \"f1_stderr\"\ : 2.2973574047539685e-05,\n \"acc\": 0.24664561957379638,\n \"acc_stderr\"\ : 0.0070256103461651745\n },\n \"harness|drop|3\": {\n \"em\": 0.0,\n\ \ \"em_stderr\": 0.0,\n \"f1\": 3.3557046979865775e-05,\n \"\ f1_stderr\": 2.2973574047539685e-05\n },\n \"harness|gsm8k|5\": {\n \ \ \"acc\": 0.0,\n \"acc_stderr\": 0.0\n },\n \"harness|winogrande|5\"\ : {\n \"acc\": 0.49329123914759276,\n \"acc_stderr\": 0.014051220692330349\n\ \ }\n}\n```" repo_url: https://huggingface.co/FabbriSimo01/GPT_Large_Quantized leaderboard_url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard point_of_contact: clementine@hf.co configs: - config_name: harness_drop_3 data_files: - split: 2023_09_23T20_31_12.168542 path: - '**/details_harness|drop|3_2023-09-23T20-31-12.168542.parquet' - split: latest path: - '**/details_harness|drop|3_2023-09-23T20-31-12.168542.parquet' - config_name: harness_gsm8k_5 data_files: - split: 2023_09_23T20_31_12.168542 path: - '**/details_harness|gsm8k|5_2023-09-23T20-31-12.168542.parquet' - split: latest path: - '**/details_harness|gsm8k|5_2023-09-23T20-31-12.168542.parquet' - config_name: harness_winogrande_5 data_files: - split: 2023_09_23T20_31_12.168542 path: - '**/details_harness|winogrande|5_2023-09-23T20-31-12.168542.parquet' - split: latest path: - '**/details_harness|winogrande|5_2023-09-23T20-31-12.168542.parquet' - config_name: results data_files: - split: 2023_09_23T20_31_12.168542 path: - results_2023-09-23T20-31-12.168542.parquet - split: latest path: - results_2023-09-23T20-31-12.168542.parquet --- # Dataset Card for Evaluation run of FabbriSimo01/GPT_Large_Quantized ## Dataset Description - **Homepage:** - **Repository:** https://huggingface.co/FabbriSimo01/GPT_Large_Quantized - **Paper:** - **Leaderboard:** https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard - **Point of Contact:** clementine@hf.co ### Dataset Summary Dataset automatically created during the evaluation run of model [FabbriSimo01/GPT_Large_Quantized](https://huggingface.co/FabbriSimo01/GPT_Large_Quantized) on the [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard). The dataset is composed of 3 configuration, each one coresponding to one of the evaluated task. The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest results. An additional configuration "results" store all the aggregated results of the run (and is used to compute and display the agregated metrics on the [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)). To load the details from a run, you can for instance do the following: ```python from datasets import load_dataset data = load_dataset("open-llm-leaderboard/details_FabbriSimo01__GPT_Large_Quantized", "harness_winogrande_5", split="train") ``` ## Latest results These are the [latest results from run 2023-09-23T20:31:12.168542](https://huggingface.co/datasets/open-llm-leaderboard/details_FabbriSimo01__GPT_Large_Quantized/blob/main/results_2023-09-23T20-31-12.168542.json)(note that their might be results for other tasks in the repos if successive evals didn't cover the same tasks. You find each in the results and the "latest" split for each eval): ```python { "all": { "em": 0.0, "em_stderr": 0.0, "f1": 3.3557046979865775e-05, "f1_stderr": 2.2973574047539685e-05, "acc": 0.24664561957379638, "acc_stderr": 0.0070256103461651745 }, "harness|drop|3": { "em": 0.0, "em_stderr": 0.0, "f1": 3.3557046979865775e-05, "f1_stderr": 2.2973574047539685e-05 }, "harness|gsm8k|5": { "acc": 0.0, "acc_stderr": 0.0 }, "harness|winogrande|5": { "acc": 0.49329123914759276, "acc_stderr": 0.014051220692330349 } } ``` ### Supported Tasks and Leaderboards [More Information Needed] ### Languages [More Information Needed] ## Dataset Structure ### Data Instances [More Information Needed] ### Data Fields [More Information Needed] ### Data Splits [More Information Needed] ## Dataset Creation ### Curation Rationale [More Information Needed] ### Source Data #### Initial Data Collection and Normalization [More Information Needed] #### Who are the source language producers? [More Information Needed] ### Annotations #### Annotation process [More Information Needed] #### Who are the annotators? [More Information Needed] ### Personal and Sensitive Information [More Information Needed] ## Considerations for Using the Data ### Social Impact of Dataset [More Information Needed] ### Discussion of Biases [More Information Needed] ### Other Known Limitations [More Information Needed] ## Additional Information ### Dataset Curators [More Information Needed] ### Licensing Information [More Information Needed] ### Citation Information [More Information Needed] ### Contributions [More Information Needed]

提供机构：

open-llm-leaderboard-old

原始信息汇总

数据集卡片 for Evaluation run of FabbriSimo01/GPT_Large_Quantized

数据集描述

数据集概述

数据集是在模型 FabbriSimo01/GPT_Large_Quantized 在 Open LLM Leaderboard 上的评估运行期间自动创建的。

数据集由3个配置组成，每个配置对应一个评估任务。

数据集从1次运行中创建。每次运行可以在每个配置中作为一个特定的分割找到，分割名称使用运行的时间戳。"train" 分割始终指向最新的结果。

一个额外的配置 "results" 存储所有运行结果的聚合（用于计算和显示 Open LLM Leaderboard 上的聚合指标）。

数据集结构

配置

harness_drop_3
- 分割: 2023_09_23T20_31_12.168542
  - 路径: **/details_harness|drop|3_2023-09-23T20-31-12.168542.parquet
- 分割: latest
  - 路径: **/details_harness|drop|3_2023-09-23T20-31-12.168542.parquet
harness_gsm8k_5
- 分割: 2023_09_23T20_31_12.168542
  - 路径: **/details_harness|gsm8k|5_2023-09-23T20-31-12.168542.parquet
- 分割: latest
  - 路径: **/details_harness|gsm8k|5_2023-09-23T20-31-12.168542.parquet
harness_winogrande_5
- 分割: 2023_09_23T20_31_12.168542
  - 路径: **/details_harness|winogrande|5_2023-09-23T20-31-12.168542.parquet
- 分割: latest
  - 路径: **/details_harness|winogrande|5_2023-09-23T20-31-12.168542.parquet
results
- 分割: 2023_09_23T20_31_12.168542
  - 路径: results_2023-09-23T20-31-12.168542.parquet
- 分割: latest
  - 路径: results_2023-09-23T20-31-12.168542.parquet

搜集汇总

数据集介绍

构建方式

在大规模语言模型评估领域，Open LLM Leaderboard 为模型性能的标准化评测提供了重要基准。该数据集源自对 FabbriSimo01/GPT_Large_Quantized 模型的一次完整评估运行，由 Hugging Face 的 Open LLM Leaderboard 平台在评测过程中自动生成。数据集包含三个配置，分别对应 DROP、GSM8K 和 WinoGrande 三项评测任务，每个配置均以时间戳命名的分割存储单次运行结果，其中“latest”分割始终指向最新数据。此外，另设“results”配置汇总全局聚合指标，用于计算并展示排行榜上的综合得分。数据以 Parquet 格式存储，确保了高效存取与可复现性。

特点

该数据集的核心特色在于其结构化的多任务评估体系与清晰的版本追踪能力。每个配置独立对应一项具体任务，便于研究者针对单一能力进行深入分析。时间戳分割的设计忠实记录了每一次评估的原始细节，支持历史结果的回溯与对比，而“latest”分割则自动指向最新运行，简化了持续追踪模型性能变化的流程。聚合结果配置则提供了诸如准确率、精确匹配、F1 分数及其标准误差等关键指标，全面反映了模型在不同维度上的表现，尤其凸显了其在 WinoGrande 任务上约 49.3% 的准确率与在 DROP、GSM8K 任务上的挑战。

使用方法

研究者可通过 Hugging Face 的 datasets 库便捷地加载该数据集，以复现评测结果或进行后续分析。加载时需指定配置名称，例如使用“harness_winogrande_5”获取 WinoGrande 任务数据，并选择“train”分割以获取最新结果。数据加载后，可结合 Pandas 等工具对 Parquet 格式的字段进行解析，提取模型在每项任务上的详细输出与性能指标。此外，通过访问“results”配置，用户能够直接获取所有任务的聚合统计，便于快速评估模型在 Open LLM Leaderboard 上的综合排名，或与其他模型进行横向对比研究。

背景与挑战

背景概述

在大型语言模型（LLM）蓬勃发展的浪潮中，如何系统性地评估模型性能成为推动技术进步的关键环节。Open LLM Leaderboard作为HuggingFace社区于2023年推出的权威评测平台，旨在通过标准化任务集衡量不同模型的泛化能力。该数据集诞生于2023年9月，由HuggingFace团队（联络人Clementine）主导创建，核心研究问题聚焦于量化评估FabbriSimo01/GPT_Large_Quantized这一量化模型在多项自然语言理解与推理任务上的表现。通过记录模型在DROP（数值推理）、GSM8K（数学推理）和WinoGrande（常识推理）三项基准上的详细结果，该数据集为社区提供了可复现的评估基线，尤其对量化模型的实际效能验证具有里程碑意义，推动了高效部署场景下模型选择的科学化进程。

当前挑战

该数据集所解决的领域挑战主要在于大型语言模型评估的标准化与可复现性困境。传统评测常因任务配置、随机种子等差异导致结果难以比较，而该数据集通过固定任务格式与评估流程（如GSM8K的5-shot设置），确保了跨模型对比的公平性。在构建过程中，挑战尤为突出：首先，量化模型（如GPT_Large_Quantized）在数值精度降低后，于DROP等需要精确计算的场景中表现极不稳定（F1值趋近于0），这要求评估框架能捕捉细微的性能退化；其次，任务结果需以时间戳分割的形式存储，并动态更新“latest”分片，对数据版本管理提出了高一致性要求；最后，面对GSM8K上完全零准确率的极端情况，如何设计鲁棒的统计指标（如附带标准误差）以反映模型真实能力，成为平衡严谨性与实用性的核心难题。

常用场景

经典使用场景

该数据集作为Open LLM Leaderboard评估流程的副产品，经典应用在于为大语言模型（LLM）提供细粒度的多任务评测基准。它记录了模型FabbriSimo01/GPT_Large_Quantized在winogrande、GSM8K和DROP三项任务上的逐样本表现，涵盖准确率、精确匹配与F1分数等关键指标。研究者可借此剖析量化模型在常识推理、数学推理与阅读理解中的能力边界，尤其适用于对比不同量化策略对模型性能的侵蚀程度。

衍生相关工作

该数据集衍生的相关工作集中于开放LLM评估框架的标准化建设。其结构化存储模式启发了后续研究者构建自动化评测流水线，例如基于parquet格式的批量结果聚合与可视化工具。此外，该数据集的发布促进了量化模型与全精度模型的横向对比研究，催生了针对特定任务（如Winogrande上的代词消解）的量化敏感性分析，为低比特模型的设计迭代提供了数据驱动的改进方向。

数据集最近研究