open-llm-leaderboard-old/details_BlouseJury__Mistral-7B-Discord-0.1-DPO

该数据集是在评估模型BlouseJury/Mistral-7B-Discord-0.1-DPO时自动创建的，包含63个配置，每个配置对应一个评估任务。数据集由1次运行生成，每次运行的结果存储为特定的分割，分割名称使用运行的时间戳。train分割始终指向最新的结果。此外，results配置存储了所有运行的聚合结果，用于计算和显示在Open LLM Leaderboard上的聚合指标。

原始信息汇总

数据集概述

数据集摘要

该数据集是在评估模型 BlouseJury/Mistral-7B-Discord-0.1-DPO 在 Open LLM Leaderboard 上的运行过程中自动创建的。

数据集组成

• 该数据集包含 63 个配置，每个配置对应一个评估任务。
• 数据集从 1 次运行中创建，每次运行可以在每个配置中找到特定的分割，分割名称使用运行的时间戳。
• "train" 分割始终指向最新的结果。
• 一个额外的配置 "results" 存储所有运行的聚合结果，用于计算和显示 Open LLM Leaderboard 上的聚合指标。

数据加载示例

python from datasets import load_dataset data = load_dataset("open-llm-leaderboard/details_BlouseJury__Mistral-7B-Discord-0.1-DPO", "harness_winogrande_5", split="train")

最新结果

以下是 2024-02-01T17:00:23.691484 运行的最新结果：

python { "all": { "acc": 0.6233951979610222, "acc_stderr": 0.032739541113838276, "acc_norm": 0.6297936917040108, "acc_norm_stderr": 0.033418584464628434, "mc1": 0.38922888616891066, "mc1_stderr": 0.017068552680690328, "mc2": 0.5527536910345616, "mc2_stderr": 0.015269414074864143 }, "harness|arc:challenge|25": { "acc": 0.6015358361774744, "acc_stderr": 0.014306946052735563, "acc_norm": 0.6322525597269625, "acc_norm_stderr": 0.014090995618168484 }, "harness|hellaswag|10": { "acc": 0.6394144592710616, "acc_stderr": 0.004791890625834196, "acc_norm": 0.8327026488747261, "acc_norm_stderr": 0.003724783389253327 }, "harness|hendrycksTest-abstract_algebra|5": { "acc": 0.29, "acc_stderr": 0.045604802157206845, "acc_norm": 0.29, "acc_norm_stderr": 0.045604802157206845 }, "harness|hendrycksTest-anatomy|5": { "acc": 0.6074074074074074, "acc_stderr": 0.0421850621536888, "acc_norm": 0.6074074074074074, "acc_norm_stderr": 0.0421850621536888 }, "harness|hendrycksTest-astronomy|5": { "acc": 0.6513157894736842, "acc_stderr": 0.0387813988879761, "acc_norm": 0.6513157894736842, "acc_norm_stderr": 0.0387813988879761 }, "harness|hendrycksTest-business_ethics|5": { "acc": 0.56, "acc_stderr": 0.04988876515698589, "acc_norm": 0.56, "acc_norm_stderr": 0.04988876515698589 }, "harness|hendrycksTest-clinical_knowledge|5": { "acc": 0.6830188679245283, "acc_stderr": 0.02863723563980089, "acc_norm": 0.6830188679245283, "acc_norm_stderr": 0.02863723563980089 }, "harness|hendrycksTest-college_biology|5": { "acc": 0.7152777777777778, "acc_stderr": 0.037738099906869334, "acc_norm": 0.7152777777777778, "acc_norm_stderr": 0.037738099906869334 }, "harness|hendrycksTest-college_chemistry|5": { "acc": 0.5, "acc_stderr": 0.050251890762960605, "acc_norm": 0.5, "acc_norm_stderr": 0.050251890762960605 }, "harness|hendrycksTest-college_computer_science|5": { "acc": 0.49, "acc_stderr": 0.05024183937956912, "acc_norm": 0.49, "acc_norm_stderr": 0.05024183937956912 }, "harness|hendrycksTest-college_mathematics|5": { "acc": 0.42, "acc_stderr": 0.049604496374885836, "acc_norm": 0.42, "acc_norm_stderr": 0.049604496374885836 }, "harness|hendrycksTest-college_medicine|5": { "acc": 0.5953757225433526, "acc_stderr": 0.03742461193887249, "acc_norm": 0.5953757225433526, "acc_norm_stderr": 0.03742461193887249 }, "harness|hendrycksTest-college_physics|5": { "acc": 0.4117647058823529, "acc_stderr": 0.04897104952726366, "acc_norm": 0.4117647058823529, "acc_norm_stderr": 0.04897104952726366 }, "harness|hendrycksTest-computer_security|5": { "acc": 0.76, "acc_stderr": 0.04292346959909283, "acc_norm": 0.76, "acc_norm_stderr": 0.04292346959909283 }, "harness|hendrycksTest-conceptual_physics|5": { "acc": 0.5787234042553191, "acc_stderr": 0.03227834510146268, "acc_norm": 0.5787234042553191, "acc_norm_stderr": 0.03227834510146268 }, "harness|hendrycksTest-econometrics|5": { "acc": 0.4473684210526316, "acc_stderr": 0.04677473004491199, "acc_norm": 0.4473684210526316, "acc_norm_stderr": 0.04677473004491199 }, "harness|hendrycksTest-electrical_engineering|5": { "acc": 0.5586206896551724, "acc_stderr": 0.04137931034482758, "acc_norm": 0.5586206896551724, "acc_norm_stderr": 0.04137931034482758 }, "harness|hendrycksTest-elementary_mathematics|5": { "acc": 0.41798941798941797, "acc_stderr": 0.025402555503260912, "acc_norm": 0.41798941798941797, "acc_norm_stderr": 0.025402555503260912 }, "harness|hendrycksTest-formal_logic|5": { "acc": 0.4444444444444444, "acc_stderr": 0.04444444444444449, "acc_norm": 0.4444444444444444, "acc_norm_stderr": 0.04444444444444449 }, "harness|hendrycksTest-global_facts|5": { "acc": 0.37, "acc_stderr": 0.048523658709391, "acc_norm": 0.37, "acc_norm_stderr": 0.048523658709391 }, "harness|hendrycksTest-high_school_biology|5": { "acc": 0.7548387096774194, "acc_stderr": 0.02447224384089553, "acc_norm": 0.7548387096774194, "acc_norm_stderr": 0.02447224384089553 }, "harness|hendrycksTest-high_school_chemistry|5": { "acc": 0.5024630541871922, "acc_stderr": 0.035179450386910616, "acc_norm": 0.5024630541871922, "acc_norm_stderr": 0.035179450386910616 }, "harness|hendrycksTest-high_school_computer_science|5": { "acc": 0.66, "acc_stderr": 0.04760952285695237, "acc_norm": 0.66, "acc_norm_stderr": 0.04760952285695237 }, "harness|hendrycksTest-high_school_european_history|5": { "acc": 0.7393939393939394, "acc_stderr": 0.034277431758165236, "acc_norm": 0.7393939393939394, "acc_norm_stderr": 0.034277431758165236 }, "harness|hendrycksTest-high_school_geography|5": { "acc": 0.803030303030303, "acc_stderr": 0.028335609732463362, "acc_norm": 0.803030303030303, "acc_norm_stderr": 0.028335609732463362 }, "harness|hendrycksTest-high_school_government_and_politics|5": { "acc": 0.8601036269430051, "acc_stderr": 0.02503387058301518, "acc_norm": 0.8601036269430051, "acc_norm_stderr": 0.02503387058301518 }, "harness|hendrycksTest-high_school_macroeconomics|5": { "acc": 0.6333333333333333, "acc_stderr": 0.02443301646605246, "acc_norm": 0.6333333333333333, "acc_norm_stderr": 0.02443301646605246 }, "harness|hendrycksTest-high_school_mathematics|5": { "acc": 0.3074074074074074, "acc_stderr": 0.028133252578815642, "acc_norm": 0.3074074074074074, "acc_norm_stderr": 0.028133252578815642 }, "harness|hendrycksTest-high_school_microeconomics|5": { "acc": 0.6470588235294118, "acc_stderr": 0.031041941304059278, "acc_norm": 0.6470588235294118, "acc_norm_stderr": 0.031041941304059278 }, "harness|hendrycksTest-high_school_physics|5": { "acc": 0.3576158940397351, "acc_stderr": 0.03913453431177258, "acc_norm": 0.3576158940397351, "acc_norm_stderr": 0.03913453431177258 }, "harness|hendrycksTest-high_school_psychology|5": { "acc": 0.8165137614678899, "acc_stderr": 0.016595259710399327, "acc_norm": 0.8165137614678899, "acc_norm_stderr": 0.016595259710399327 }, "harness|hendrycksTest-high_school_statistics|5": { "acc": 0.5509259259259259, "acc_stderr": 0.03392238405321617, "acc_norm": 0.5509259259259259, "acc_norm_stderr": 0.03392238405321617 }, "harness|hendrycksTest-high_school_us_history|5": { "acc": 0.8088235294117647, "acc_stderr": 0.02759917430064076, "acc_norm": 0.8088235294117647, "acc_norm_stderr": 0.02759917430064076 }, "harness|hendrycksTest-high_school_world_history|5": { "acc": 0.7848101265822784, "acc_stderr": 0.026750826994676173, "acc_norm": 0.7848101265822784, "acc_norm_stderr": 0.026750826994676173 }, "harness|hendrycksTest-human_aging|5": { "acc": 0.6681614349775785, "acc_stderr": 0.03160295143776678, "acc_norm": 0.6681614349775785, "acc_norm_stderr": 0.03160295143776678 }, "harness|hendrycksTest-human_sexuality|5": { "acc": 0.7633587786259542, "acc_stderr": 0.03727673575596913, "acc_norm": 0.7633587786259542, "acc_norm_stderr": 0.03727673575596913 }, "harness|hendrycksTest-international_law|5": { "acc": 0.7603305785123967, "acc_stderr": 0.03896878985070417, "acc_norm": 0.7603305785123967, "acc_norm_stderr": 0.03896878985070417 }, "harness|hendrycksTest-jurisprudence|5": { "acc": 0.7592592592592593, "acc_stderr": 0.041331194402438376, "acc_norm": 0.7592592592592593, "acc_norm_stderr": 0.041331194402438376 }, "harness|hendrycksTest-logical_fallacies|5": { "acc": 0.7423312883435583, "acc_stderr": 0.03436150827846917, "acc_norm": 0.7423312883435583, "acc_norm_stderr": 0.03436150827846917 }, "harness|hendrycksTest-machine_learning|5": { "acc": 0.38392857142857145, "acc_stderr": 0.04616143075028547, "acc_norm": 0.38392857142857145, "acc_norm_stderr": 0.04616143075028547 }, "harness|hendrycksTest-management|5": { "acc": 0.7669902912621359, "acc_stderr": 0.04185832598928315, "acc_norm": 0.7669902912621359, "acc_norm_stderr": 0.04185832598928315 }, "harness|hendrycksTest-marketing|5": { "acc": 0.8675213675213675, "acc_stderr": 0.022209309073165616, "acc_norm": 0.8675213675213675, "acc_norm_stderr": 0.022209309073165616 }, "harness|hendrycksTest-medical_genetics|5": { "acc": 0.69, "acc_stderr": 0.04648231987117316, "acc_norm": 0.69, "acc_norm_stderr": 0.04648231987117316 }, "harness|hendrycksTest-miscellaneous|5": { "acc": 0.8109833971902938, "acc_stderr": 0.014000791294407003, "acc_norm": 0.8109833971902938, "acc_norm_stderr": 0.014000791294407003 }, "harness|hendrycksTest-moral_disputes|5": { "acc": 0.7023121387283237, "acc_stderr": 0.024617055388676992, "acc_norm": 0.7023121387283237, "acc_norm_stderr": 0.024617055388676992 }, "harness|hendrycksTest-moral_scenarios|5": { "acc": 0.3396648044692737, "acc_stderr": 0.015839400406212505, "acc_norm": 0.3396648044692737, "acc_norm_stderr": 0.015839400406212505 }, "harness|hendrycksTest-nutrition|5": { "acc": 0.7091503267973857, "acc_stderr": 0.02600480036395213, "acc_norm": 0.7091503267973857, "acc_norm_stderr": 0.02600480036395213 }, "harness|hendrycksTest-philosophy|5": { "acc": 0.707395498392283, "acc_stderr": 0.02583989833487798, "acc_norm": 0.707395498392283, "acc_norm_stderr": 0.02583989833487798 }, "harness|hendrycksTest-prehistory|5": { "acc": 0.7283950617283951, "acc_stderr": 0.02474862449053738, "acc_norm": 0.7283950617283951, "acc_norm_stderr": 0.02474862449053738 }, "harness|hendrycksTest-professional_accounting|5": { "acc": 0.46808510638297873, "acc_stderr": 0.029766675075873866, "acc_norm": 0.46808510638297873, "acc_norm_stderr": 0.029766675075873866 }, "harness|hendrycksTest-professional_law|5": { "acc": 0.439374185136897, "acc_stderr": 0.012676014778580214, "acc_norm": 0.439374185136897, "acc_norm_stderr": 0.012676014778580214 }, "harness|hendrycksTest-professional_medicine|5": { "acc": 0.6764705882352942, "acc_stderr": 0.028418208619406755, "acc_norm": 0.6764705882352942, "acc_norm_stderr": 0.028418208619406755 }, "harness|hendrycksTest-professional_psychology|5": { "acc": 0.6339869281045751, "acc_stderr": 0.019488025745529675, "acc_norm": 0.6339869281045751, "acc_norm_stderr": 0.019488025745529675 }, "harness|hendrycksTest-public_relations|5": { "acc": 0.6454545454545455, "acc_stderr": 0.045820048415054174, "acc_norm": 0.6454545454545455, "acc_norm_stderr": 0.045820048415054174 }, "harness|hendrycksTest-security_studies|5": { "acc": 0.7142857142857143, "acc_stderr": 0.0289205832206756, "acc_norm": 0.7142857142857143, "acc_norm_stderr": 0.0289205832206756 }, "harness|hendrycksTest-sociology|5": { "acc": 0.8308457711442786, "acc_stderr": 0.026508590656233264, "acc_norm": 0.8308457711442786, "acc_norm_stderr": 0.026508590656233264 }, "harness|hendrycksTest-us_foreign_policy|5": { "acc": 0.82, "acc_stderr": 0.038612291966536955, "acc_norm": 0.82, "acc_norm_stderr": 0.038612291966536955 }, "harness|hendrycksTest-virology|5": { "acc": 0.5240963855421686, "acc_stderr": 0.03887971849597264, "acc_norm": 0.5240963855421686, "acc_norm_stderr": 0.03887971849597264 }, "harness|hendrycksTest-world_religions|5": { "acc": 0.8128654970760234, "acc_stderr": 0.029913127232368036, "acc_norm": 0.8128654970760234, "acc_norm_stderr": 0.029913127232368036 }, "harness|truthfulqa:mc|0": { "mc1": 0.38922888616891066, "mc1_stderr": 0.017068552680690328, "mc2": 0.5527536910345616, "mc2_stderr": 0.015269414074864143 }, "harness|winogrande|5": { "acc": 0.7892659826361483, "acc_stderr": 0.011462046419710681 }, "harness|gsm8k|5": { "acc": 0.30401819560272936, "acc_stderr": 0.012670420440198654 } }

搜集汇总

数据集介绍

构建方式

该数据集是在Open LLM Leaderboard框架下，针对BlouseJury/Mistral-7B-Discord-0.1-DPO模型进行自动化评估过程中生成的。数据集由63个配置组成，每个配置对应一项被评估的任务，例如ARC挑战、HellaSwag、GSM8K及涵盖多学科的Hendrycks测试等。每次评估运行均作为一个独立的分片存储于各配置中，分片名称以运行时间戳命名，而“train”分片始终指向最新一次评估的结果。此外，一个名为“results”的附加配置汇总了所有运行的聚合指标，用于在排行榜上计算和展示综合性能。

特点

数据集的核心特点在于其结构化的评估记录方式，能够完整追踪模型在63项不同任务上的表现。每项任务的配置下均包含多个时间戳分片，便于研究者对比模型在不同时间点的性能变化。数据集中存储了详细的评估指标，如准确率（acc）、标准化准确率（acc_norm）及其标准误差，以及TruthfulQA中的多选指标（mc1、mc2）等，为模型能力分析提供了丰富的维度。这种设计使得数据集不仅是评估结果的存档，更是模型性能演进的忠实记录。

使用方法

研究者可通过HuggingFace的datasets库加载该数据集，指定具体的任务配置（如“harness_winogrande_5”）和分片名称（如“train”或时间戳分片）来获取详细评估数据。例如，使用load_dataset函数并传入数据集名称、配置名及split参数即可加载特定任务的评估结果。对于需要分析模型整体表现的场景，可直接访问“results”配置，其中包含了所有任务的聚合指标，便于进行横向比较和综合评估。

背景与挑战

背景概述

在大语言模型（LLM）领域，模型性能的客观评估是推动技术发展的关键环节。Open LLM Leaderboard由Hugging Face团队于2023年发起，旨在通过标准化基准测试，为社区提供透明、可复现的模型比较平台。该数据集记录了BlouseJury/Mistral-7B-Discord-0.1-DPO模型在63项任务上的评估结果，涵盖常识推理（如HellaSwag）、数学推理（如GSM8K）及多学科知识（如MMLU）等维度。作为Leaderboard的衍生数据，它聚焦于Mistral-7B经直接偏好优化（DPO）微调后的表现，揭示了模型在标准化评估框架下的综合能力与局限性，为后续模型迭代与训练策略优化提供了实证基础。

当前挑战

该数据集所解决的领域问题在于，大语言模型性能的碎片化评估常导致结论偏差，而标准化基准（如ARC、WinoGrande）能系统衡量模型在推理、知识检索与对抗性测试中的鲁棒性。构建过程中面临的核心挑战包括：一是任务异构性——63项配置覆盖从中学数学到专业医学的跨领域知识，需确保评估逻辑的一致性；二是结果可复现性——自动生成的运行记录需精确关联时间戳与配置版本，避免因模型更新或环境差异引入噪声；三是统计显著性——如GSM8K任务中30.4%的准确率与HellaSwag中83.3%的归一化准确率差异，要求对标准误差（如0.0127）进行严谨解释，以区分模型能力与随机波动的影响。

常用场景

经典使用场景

在大型语言模型的评估生态中，该数据集作为Open LLM Leaderboard的自动生成产物，其核心用途在于系统性地记录和追踪模型BlouseJury/Mistral-7B-Discord-0.1-DPO在63个多样化任务上的细粒度表现。它整合了来自ARC-Challenge、HellaSwag、GSM8K等经典基准的评估结果，并提供了标准化接口以便研究者加载特定任务的运行详情，从而支持对模型推理、常识理解和数学能力等维度的深入剖析。

解决学术问题

该数据集解决了大模型评估中结果可复现性与透明度不足的学术难题。通过将每一次评估运行的原始指标（包括准确率、标准差及归一化分数）以结构化方式存储，它使得跨模型、跨时间点的性能对比成为可能，并助力研究者剖析模型在科学、人文、伦理等57个MMLU子领域中的知识短板，从而为理解模型能力边界和指导后续优化提供了坚实的数据基础。

衍生相关工作

该数据集衍生了一系列围绕大模型评估标准化与自动化的经典工作。基于其提供的多任务细粒度结果，研究者得以构建模型能力雷达图、开发性能趋势分析工具，并推动评估框架如EleutherAI LM Evaluation Harness的广泛应用。此外，该数据集作为Open LLM Leaderboard的核心组件，直接催生了大量关于模型泛化能力、训练技巧效果验证以及开源模型横向对比的实证研究，加速了LLM领域的透明化进程。