open-llm-leaderboard-old/details_NousResearch__Nous-Hermes-2-Mistral-7B-DPO

该数据集是在评估模型NousResearch/Nous-Hermes-2-Mistral-7B-DPO时自动生成的，包含63个配置，每个配置对应一个评估任务。数据集由2次运行生成，每次运行的结果存储为特定的分割，分割名称使用运行的时间戳。train分割始终指向最新的结果。此外，还有一个名为results的配置，存储了所有运行的聚合结果，用于计算和展示在Open LLM Leaderboard上的聚合指标。

原始信息汇总

数据集概述

数据集摘要

该数据集是在模型 NousResearch/Nous-Hermes-2-Mistral-7B-DPO 在 Open LLM Leaderboard 上的评估运行期间自动创建的。

数据集组成

• 数据集包含 63 个配置，每个配置对应一个评估任务。
• 数据集从 2 次运行中创建，每次运行可以在每个配置中作为一个特定的分割找到，分割名称使用运行的时间戳。
• "train" 分割始终指向最新的结果。
• 一个额外的配置 "results" 存储所有运行的聚合结果，用于计算和显示 Open LLM Leaderboard 上的聚合指标。

数据加载示例

python from datasets import load_dataset data = load_dataset("open-llm-leaderboard/details_NousResearch__Nous-Hermes-2-Mistral-7B-DPO", "harness_winogrande_5", split="train")

最新结果

以下是 2024-02-21T00:48:36.188963 运行的最新结果：

python { "all": { "acc": 0.6347060781129819, "acc_stderr": 0.03219733375318615, "acc_norm": 0.63654655633088, "acc_norm_stderr": 0.03283944068311936, "mc1": 0.38555691554467564, "mc1_stderr": 0.017038839010591667, "mc2": 0.5578131545614541, "mc2_stderr": 0.015293843998956788 }, "harness|arc:challenge|25": { "acc": 0.6168941979522184, "acc_stderr": 0.014206472661672876, "acc_norm": 0.6569965870307167, "acc_norm_stderr": 0.013872423223718164 }, "harness|hellaswag|10": { "acc": 0.6608245369448317, "acc_stderr": 0.004724619193427586, "acc_norm": 0.8494323839872535, "acc_norm_stderr": 0.003568960247101678 }, "harness|hendrycksTest-abstract_algebra|5": { "acc": 0.28, "acc_stderr": 0.045126085985421276, "acc_norm": 0.28, "acc_norm_stderr": 0.045126085985421276 }, "harness|hendrycksTest-anatomy|5": { "acc": 0.5851851851851851, "acc_stderr": 0.04256193767901408, "acc_norm": 0.5851851851851851, "acc_norm_stderr": 0.04256193767901408 }, "harness|hendrycksTest-astronomy|5": { "acc": 0.6842105263157895, "acc_stderr": 0.0378272898086547, "acc_norm": 0.6842105263157895, "acc_norm_stderr": 0.0378272898086547 }, "harness|hendrycksTest-business_ethics|5": { "acc": 0.58, "acc_stderr": 0.049604496374885836, "acc_norm": 0.58, "acc_norm_stderr": 0.049604496374885836 }, "harness|hendrycksTest-clinical_knowledge|5": { "acc": 0.6867924528301886, "acc_stderr": 0.028544793319055326, "acc_norm": 0.6867924528301886, "acc_norm_stderr": 0.028544793319055326 }, "harness|hendrycksTest-college_biology|5": { "acc": 0.7708333333333334, "acc_stderr": 0.03514697467862388, "acc_norm": 0.7708333333333334, "acc_norm_stderr": 0.03514697467862388 }, "harness|hendrycksTest-college_chemistry|5": { "acc": 0.46, "acc_stderr": 0.05009082659620332, "acc_norm": 0.46, "acc_norm_stderr": 0.05009082659620332 }, "harness|hendrycksTest-college_computer_science|5": { "acc": 0.46, "acc_stderr": 0.05009082659620333, "acc_norm": 0.46, "acc_norm_stderr": 0.05009082659620333 }, "harness|hendrycksTest-college_mathematics|5": { "acc": 0.24, "acc_stderr": 0.04292346959909282, "acc_norm": 0.24, "acc_norm_stderr": 0.04292346959909282 }, "harness|hendrycksTest-college_medicine|5": { "acc": 0.6069364161849711, "acc_stderr": 0.0372424959581773, "acc_norm": 0.6069364161849711, "acc_norm_stderr": 0.0372424959581773 }, "harness|hendrycksTest-college_physics|5": { "acc": 0.39215686274509803, "acc_stderr": 0.048580835742663454, "acc_norm": 0.39215686274509803, "acc_norm_stderr": 0.048580835742663454 }, "harness|hendrycksTest-computer_security|5": { "acc": 0.75, "acc_stderr": 0.04351941398892446, "acc_norm": 0.75, "acc_norm_stderr": 0.04351941398892446 }, "harness|hendrycksTest-conceptual_physics|5": { "acc": 0.548936170212766, "acc_stderr": 0.032529096196131965, "acc_norm": 0.548936170212766, "acc_norm_stderr": 0.032529096196131965 }, "harness|hendrycksTest-econometrics|5": { "acc": 0.4649122807017544, "acc_stderr": 0.046920083813689104, "acc_norm": 0.4649122807017544, "acc_norm_stderr": 0.046920083813689104 }, "harness|hendrycksTest-electrical_engineering|5": { "acc": 0.5310344827586206, "acc_stderr": 0.04158632762097828, "acc_norm": 0.5310344827586206, "acc_norm_stderr": 0.04158632762097828 }, "harness|hendrycksTest-elementary_mathematics|5": { "acc": 0.42328042328042326, "acc_stderr": 0.02544636563440678, "acc_norm": 0.42328042328042326, "acc_norm_stderr": 0.02544636563440678 }, "harness|hendrycksTest-formal_logic|5": { "acc": 0.42857142857142855, "acc_stderr": 0.04426266681379909, "acc_norm": 0.42857142857142855, "acc_norm_stderr": 0.04426266681379909 }, "harness|hendrycksTest-global_facts|5": { "acc": 0.37, "acc_stderr": 0.04852365870939099, "acc_norm": 0.37, "acc_norm_stderr": 0.04852365870939099 }, "harness|hendrycksTest-high_school_biology|5": { "acc": 0.7967741935483871, "acc_stderr": 0.02289168798455495, "acc_norm": 0.7967741935483871, "acc_norm_stderr": 0.02289168798455495 }, "harness|hendrycksTest-high_school_chemistry|5": { "acc": 0.5123152709359606, "acc_stderr": 0.035169204442208966, "acc_norm": 0.5123152709359606, "acc_norm_stderr": 0.035169204442208966 }, "harness|hendrycksTest-high_school_computer_science|5": { "acc": 0.67, "acc_stderr": 0.047258156262526066, "acc_norm": 0.67, "acc_norm_stderr": 0.047258156262526066 }, "harness|hendrycksTest-high_school_european_history|5": { "acc": 0.793939393939394, "acc_stderr": 0.03158415324047711, "acc_norm": 0.793939393939394, "acc_norm_stderr": 0.03158415324047711 }, "harness|hendrycksTest-high_school_geography|5": { "acc": 0.7828282828282829, "acc_stderr": 0.029376616484945633, "acc_norm": 0.7828282828282829, "acc_norm_stderr": 0.029376616484945633 }, "harness|hendrycksTest-high_school_government_and_politics|5": { "acc": 0.8808290155440415, "acc_stderr": 0.023381935348121434, "acc_norm": 0.8808290155440415, "acc_norm_stderr": 0.023381935348121434 }, "harness|hendrycksTest-high_school_macroeconomics|5": { "acc": 0.6051282051282051, "acc_stderr": 0.024784316942156406, "acc_norm": 0.6051282051282051, "acc_norm_stderr": 0.024784316942156406 }, "harness|hendrycksTest-high_school_mathematics|5": { "acc": 0.3111111111111111, "acc_stderr": 0.028226446749683515, "acc_norm": 0.3111111111111111, "acc_norm_stderr": 0.028226446749683515 }, "harness|hendrycksTest-high_school_microeconomics|5": { "acc": 0.6764705882352942, "acc_stderr": 0.0303883535518868, "acc_norm": 0.6764705882352942, "acc_norm_stderr": 0.0303883535518868 }, "harness|hendrycksTest-high_school_physics|5": { "acc": 0.3509933774834437, "acc_stderr": 0.03896981964257375, "acc_norm": 0.3509933774834437, "acc_norm_stderr": 0.03896981964257375 }, "harness|hendrycksTest-high_school_psychology|5": { "acc": 0.8311926605504587, "acc_stderr": 0.016060056268530336, "acc_norm": 0.8311926605504587, "acc_norm_stderr": 0.016060056268530336 }, "harness|hendrycksTest-high_school_statistics|5": { "acc": 0.49074074074074076, "acc_stderr": 0.034093869469927006, "acc_norm": 0.49074074074074076, "acc_norm_stderr": 0.034093869469927006 }, "harness|hendrycksTest-high_school_us_history|5": { "acc": 0.7941176470588235, "acc_stderr": 0.028379449451588663, "acc_norm": 0.7941176470588235, "acc_norm_stderr": 0.028379449451588663 }, "harness|hendrycksTest-high_school_world_history|5": { "acc": 0.810126582278481, "acc_stderr": 0.02553010046023349, "acc_norm": 0.810126582278481, "acc_norm_stderr": 0.02553010046023349 }, "harness|hendrycksTest-human_aging|5": { "acc": 0.7085201793721974, "acc_stderr": 0.030500283176545843, "acc_norm": 0.7085201793721974, "acc_norm_stderr": 0.030500283176545843 }, "harness|hendrycksTest-human_sexuality|5": { "acc": 0.7709923664122137, "acc_stderr": 0.036853466317118506, "acc_norm": 0.7709923664122137, "acc_norm_stderr": 0.036853466317118506 }, "harness|hendrycksTest-international_law|5": { "acc": 0.7768595041322314, "acc_stderr": 0.03800754475228732, "acc_norm": 0.7768595041322314, "acc_norm_stderr": 0.03800754475228732 }, "harness|hendrycksTest-jurisprudence|5": { "acc": 0.7777777777777778, "acc_stderr": 0.0401910747255735, "acc_norm": 0.7777777777777778, "acc_norm_stderr": 0.0401910747255735 }, "harness|hendrycksTest-logical_fallacies|5": { "acc": 0.7975460122699386, "acc_stderr": 0.031570650789119005, "acc_norm": 0.7975460122699386, "acc_norm_stderr": 0.031570650789119005 }, "harness|hendrycksTest-machine_learning|5": { "acc": 0.49107142857142855, "acc_stderr": 0.04745033255489123, "acc_norm": 0.49107142857142855, "acc_norm_stderr": 0.04745033255489123 }, "harness|hendrycksTest-management|5": { "acc": 0.7864077669902912, "acc_stderr": 0.040580420156460344, "acc_norm": 0.7864077669902912, "acc_norm_stderr": 0.040580420156460344 }, "harness|hendrycksTest-marketing|5": { "acc": 0.8632478632478633, "acc_stderr": 0.02250903393707781, "acc_norm": 0.8632478632478633, "acc_norm_stderr": 0.02250903393707781 }, "harness|hendrycksTest-medical_genetics|5": { "acc": 0.7, "acc_stderr": 0.046056618647183814, "acc_norm": 0.7, "acc_norm_stderr": 0.046056618647183814 }, "harness|hendrycksTest-miscellaneous|5": { "acc": 0.8275862068965517, "acc_stderr": 0.013507943909371803, "acc_norm": 0.8275862068965517, "acc_norm_stderr": 0.013507943909371803 }, "harness|hendrycksTest-moral_disputes|5": { "acc": 0.7138728323699421, "acc_stderr": 0.02433214677913413, "acc_norm": 0.7138728323699421, "acc_norm_stderr": 0.02433214677913413 }, "harness|hendrycksTest-moral_scenarios|5": { "acc": 0.2860335195530726, "acc_stderr": 0.015113972129062143, "acc_norm": 0.2860335195530726, "acc_norm_stderr": 0.015113972129062143 }, "harness|hendrycksTest-nutrition|5": { "acc": 0.7418300653594772, "acc_stderr": 0.02505850331695814, "acc_norm": 0.7418300653594772, "acc_norm_stderr": 0.02505850331695814 }, "harness|hendrycksTest-philosophy|5": { "acc": 0.6816720257234726, "acc_stderr": 0.026457225067811025, "acc_norm": 0.6816720257234726, "acc_norm_stderr": 0.026457225067811025 }, "harness|hendrycksTest-prehistory|5": { "acc": 0.75, "acc_stderr": 0.02409347123262133, "acc_norm": 0.75, "acc_norm_stderr": 0.02409347123262133 }, "harness|hendrycksTest-professional_accounting|5": { "acc": 0.5070921985815603, "acc_stderr": 0.02982449855912901, "acc_norm": 0.5070921985815603, "acc_norm_stderr": 0.02982449855912901 }, "harness|hendrycksTest-professional_law|5": { "acc": 0.46740547588005216, "acc_stderr": 0.012743072942653342, "acc_norm": 0.46740547588005216, "acc_norm_stderr": 0.012743072942653342 }, "harness|hendrycksTest-professional_medicine|5": { "acc": 0.6727941176470589, "acc_stderr": 0.02850145286039655, "acc_norm": 0.6727941176470589, "acc_norm_stderr": 0.02850145286039655 }, "harness|hendrycksTest-professional_psychology|5": { "acc": 0.6715686274509803, "acc_stderr": 0.018999707383162666, "acc_norm": 0.6715686274509803, "acc_norm_stderr": 0.018999707383162666 }, "harness|hendrycksTest-public_relations|5": { "acc": 0.6727272727272727, "acc_stderr": 0.04494290866252091, "acc_norm": 0.6727272727272727, "acc_norm_stderr": 0.04494290866252091 }, "harness|hendrycksTest-security_studies|5": { "acc": 0.7387755102040816, "acc_stderr": 0.028123429335142773, "acc_norm": 0.7387755102040816, "acc_norm_stderr": 0.028123429335142773 }, "harness|hendrycksTest-sociology|5": { "acc": 0.8059701492537313, "acc_stderr": 0.027962677604768914, "acc_norm": 0.8059701492537313, "acc_norm_stderr": 0.027962677604768914 }, "harness|hendrycksTest-us_foreign_policy|5": { "acc": 0.88, "acc_stderr": 0.032659863237109066, "acc_norm": 0.88, "acc_norm_stderr": 0.032659863237109066 }, "harness|hendrycksTest-virology|5": { "acc": 0.5602409638554217, "acc_stderr": 0.03864139923699122, "acc_norm": 0.5602409638554217, "acc_norm_stderr": 0.03864139923699122 }, "harness|hendrycksTest-world_religions|5": { "acc": 0.8304093567251462, "acc_stderr": 0.02878210810540171, "acc_norm": 0.8304093567251462, "acc_norm_stderr": 0.02878210810540171 }, "harness|truthfulqa:mc|0": { "mc1": 0.38555691554467564, "mc1_stderr": 0.017038839010591667, "mc2": 0.5578131545614541, "mc2_stderr": 0.015293843998956788 }, "harness|winogrande|5": { "acc": 0.7845303867403315, "acc_stderr": 0.011555295286059282 }, "harness|gsm8k|5": { "acc": 0.6050037907505686, "acc_stderr": 0.0134653549699732 } }

配置详情

• config_name: harness_arc_challenge_25

  • 分割:
    • 2024_02_21T00_41_28.762847
      • 路径: **/details_harness|arc:challenge|25_2024-02-21T00-41-28.762847.parquet
    • 2024_02_21T00_48_36.188963
      • 路径: **/details_harness|arc:challenge|25_2024-02-21T00-48-36.188963.parquet
    • latest
      • 路径: **/details_harness|arc:challenge|25_2024-02-21T00-48-36.188963.parquet

• config_name: harness_gsm8k_5

  • 分割:
    • 2024_02_21T00_41_28.762847
      • 路径: **/details_harness|gsm8k|5_2024-02-21T00-41-28.762847.parquet
    • 2024_02_21T00_48_36.188963
      • 路径: **/details_harness|gsm8k|5_2024-02-21T00-48-36.188963.parquet
    • latest
      • 路径: **/details_harness|gsm8k|5_2024-02-21T00-48-36.188963.parquet

• config_name: harness_hellaswag_10

  • 分割:
    • 2024_02_21T00_41_28.762847
      • 路径: **/details_harness|hellaswag|10_2024-02-21T00-41-28.762847.parquet
    • 2024_02_21T00_48_36.188963
      • 路径: **/details_harness|hellaswag|10_2024-02-21T00-48-36.188963.parquet
    • latest
      • 路径: **/details_harness|hellaswag|10_2024-02-21T00-48-36.188963.parquet

• config_name: harness_hendrycksTest_5

  • 分割:
    • 2024_02_21T00_41_28.762847
      • 路径:
        • **/details_harness|hendrycksTest-abstract_algebra|5_2024-02-21T00-41-28.762847.parquet
        • **/details_harness|hendrycksTest-anatomy|5_2024-02-21T00-41-28.762847.parquet
        • **/details_harness|hendrycksTest-astronomy|5_2024-02-21T00-41-28.762847.parquet
        • **/details_harness|hendrycksTest-business_ethics|5_2024-02-21T00-41-28.762847.parquet
        • **/details_harness|hendrycksTest-clinical_knowledge|5_2024-02-21T00-41-28.762847.parquet
        • **/details_harness|hendrycksTest-college_biology|5_2024-02-21T00-41-28.762847.parquet
        • **/details_harness|hendrycksTest-college_chemistry|5_2024-02-21T00-41-28.762847.parquet
        • **/details_harness|hendrycksTest-college_computer_science|5_2024-02-21T00-41-28.762847.parquet
        • **/details_harness|hendrycksTest-college_mathematics|5_2024-02-21T00-41-28.762847.parquet
        • **/details_harness|hendrycksTest-college_medicine|5_2024-02-21T00-41-28.762847.parquet
        • **/details_harness|hendrycksTest-college_physics|5_2024-02-21T00-41-28.762847.parquet
        • **/details_harness|hendrycksTest-computer_security|5_2024-02-21T00-41-28.762847.parquet
        • **/details_harness|hendrycksTest-conceptual_physics|5_2024-02-21T00-41-28.762847.parquet
        • **/details_harness|hendrycksTest-econometrics|5_2024-02-21T00-41-28.762847.parquet

搜集汇总

数据集介绍

构建方式

在大型语言模型评测领域，该数据集是专为记录Nous-Hermes-2-Mistral-7B-DPO模型在Open LLM Leaderboard上的评估过程而自动生成的。其构建依托于两次独立的评估运行，每次运行的结果均以时间戳为标识，作为独立的切分存储于各个配置中。数据集共包含63个配置，每个配置对应一项被评估的任务，并额外设有“results”配置，用于聚合所有运行的总体指标，为排行榜上的综合评分提供计算基础。

使用方法

使用者可通过HuggingFace的datasets库便捷地加载所需数据。例如，利用`load_dataset`函数，指定数据集名称与任务配置（如"harness_winogrande_5"），并选择切分（如"latest"）即可获取最新评估的详细记录。每个切分以Parquet格式存储，便于高效读取。若要分析特定运行的历史结果，只需将切分参数替换为对应的时间戳标识即可，这一灵活性支持了纵向对比研究。

背景与挑战

背景概述

Open LLM Leaderboard 是 Hugging Face 社区于 2023 年发起的一项开源大语言模型评估基准，由 Hugging Face 团队与多位研究者共同维护，旨在系统性地衡量各类开源大语言模型在多个维度上的性能表现。该数据集记录了 NousResearch 团队开发的 Nous-Hermes-2-Mistral-7B-DPO 模型在 Leaderboard 上的完整评估结果，涵盖 ARC-Challenge、HellaSwag、MMLU、TruthfulQA、Winogrande 和 GSM8K 等 63 个多样化任务。通过精确的准确率与标准误差指标，该数据集为研究人员提供了透明、可复现的模型性能画像，推动了开源大语言模型评估的标准化进程，成为社区衡量模型能力的重要参考。

当前挑战

该数据集所面临的挑战主要体现在两个层面。在领域问题层面，大语言模型的评估需要覆盖知识推理、常识理解、数学计算、伦理判断等广泛能力，而现有基准难以全面反映模型在复杂真实场景中的表现，例如 MMLU 中专业医学与物理学任务的低准确率揭示了模型在深度领域知识上的不足。在构建过程中，评估结果的时效性与一致性是核心难题，多次运行间的结果差异、任务配置的多样性以及数据文件的版本管理均增加了结果复现与横向比较的复杂性，需要精细化的元数据标注与自动化流水线来保障评估的可靠性。

常用场景

经典使用场景

该数据集的核心用途在于为大规模语言模型在开放LLM排行榜上的性能评估提供细粒度的评测记录。其经典场景是作为模型推理能力的标准化测试平台，通过涵盖ARC挑战赛、HellaSwag、GSM8K及MMLU（涵盖从抽象代数到病毒学的57个学科）等多样化任务，系统性地衡量模型在常识推理、数学解题与多领域知识掌握上的表现。研究者可加载特定任务配置（如harness_winogrande_5）并提取最新或历史运行的分片数据，从而复现或对比模型在不同时间点的得分。

解决学术问题

该数据集解决了语言模型评估中缺乏统一、可复现的细粒度基准这一关键学术难题。通过存储每次评测的完整原始结果（包括精确率与标准误差），它使研究者能够深入分析模型在特定子任务上的优势与短板，例如区分模型在HellaSwag的归一化准确率（84.9%）与GSM8K的数学推理准确率（60.5%）之间的差异。这为诊断模型泛化能力、揭示训练策略（如直接偏好优化DPO）对特定认知维度的提升效果提供了实证基础，推动了更科学的模型对比方法论。

实际应用

在实际应用中，该数据集被广泛应用于模型选型与部署前的鲁棒性验证。企业和研究机构可依据其记录的57个MMLU学科得分，筛选出在专业领域（如临床知识、法学、物理学）表现突出的模型，从而为医疗诊断辅助、法律文书分析或教育辅导系统匹配合适的语言引擎。此外，通过追踪同一模型在不同时间点的评测结果，开发者能监控模型更新是否引入性能退化，确保生产环境中模型行为的稳定与可靠。

数据集最近研究

最新研究方向

在大型语言模型（LLM）领域，基于人类反馈的强化学习（RLHF）与直接偏好优化（DPO）已成为提升模型对齐能力的前沿技术。Nous-Hermes-2-Mistral-7B-DPO作为这一方向的代表性模型，通过Open LLM Leaderboard评估框架在63个配置任务中展现了其综合性能。该数据集记录了模型在ARC挑战、HellaSwag、MMLU及GSM8K等基准上的详细表现，其中在HellaSwag上归一化准确率达84.94%，在MMLU的政府与政治子集上达到88.08%，凸显了DPO训练在增强常识推理与专业知识掌握上的显著效果。这一评估体系不仅为模型迭代提供了量化依据，更推动了开源社区对偏好学习算法的深入探索，其影响在于为构建更安全、更符合人类意图的智能系统奠定了实证基础，成为当前LLM对齐研究的重要里程碑。