open-llm-leaderboard-old/details_fhai50032__SamChat

该数据集是在模型fhai50032/SamChat的评估运行期间自动创建的，用于在Open LLM Leaderboard上进行评估。数据集由63个配置组成，每个配置对应一个评估任务。数据集从1次运行中创建，每次运行可以在每个配置中找到，分割以运行的时间戳命名。train分割始终指向最新结果。此外，results配置存储了所有运行的聚合结果，并用于计算和显示Open LLM Leaderboard上的聚合指标。

原始信息汇总

数据集概述

该数据集是在对模型 fhai50032/SamChat 进行评估运行期间自动创建的。数据集包含 63 个配置，每个配置对应一个评估任务。数据集从 1 次运行中创建，每次运行的详细信息可以在每个配置中找到，以运行的时间戳命名的特定分片形式存储。"train" 分片始终指向最新的结果。

数据集结构

数据集包含多个配置，每个配置对应一个评估任务。以下是部分配置的示例：

• 配置名称: harness_arc_challenge_25

  • 数据文件:
    • 分片: 2024_02_11T14_43_14.190988
      • 路径: **/details_harness|arc:challenge|25_2024-02-11T14-43-14.190988.parquet
    • 分片: latest
      • 路径: **/details_harness|arc:challenge|25_2024-02-11T14-43-14.190988.parquet

• 配置名称: harness_gsm8k_5

  • 数据文件:
    • 分片: 2024_02_11T14_43_14.190988
      • 路径: **/details_harness|gsm8k|5_2024-02-11T14-43-14.190988.parquet
    • 分片: latest
      • 路径: **/details_harness|gsm8k|5_2024-02-11T14-43-14.190988.parquet

• 配置名称: harness_hellaswag_10

  • 数据文件:
    • 分片: 2024_02_11T14_43_14.190988
      • 路径: **/details_harness|hellaswag|10_2024-02-11T14-43-14.190988.parquet
    • 分片: latest
      • 路径: **/details_harness|hellaswag|10_2024-02-11T14-43-14.190988.parquet

• 配置名称: harness_hendrycksTest_5

  • 数据文件:
    • 分片: 2024_02_11T14_43_14.190988
      • 路径:
        • **/details_harness|hendrycksTest-abstract_algebra|5_2024-02-11T14-43-14.190988.parquet
        • **/details_harness|hendrycksTest-anatomy|5_2024-02-11T14-43-14.190988.parquet
        • **/details_harness|hendrycksTest-astronomy|5_2024-02-11T14-43-14.190988.parquet
        • **/details_harness|hendrycksTest-business_ethics|5_2024-02-11T14-43-14.190988.parquet
        • **/details_harness|hendrycksTest-clinical_knowledge|5_2024-02-11T14-43-14.190988.parquet
        • **/details_harness|hendrycksTest-college_biology|5_2024-02-11T14-43-14.190988.parquet
        • **/details_harness|hendrycksTest-college_chemistry|5_2024-02-11T14-43-14.190988.parquet
        • **/details_harness|hendrycksTest-college_computer_science|5_2024-02-11T14-43-14.190988.parquet
        • **/details_harness|hendrycksTest-college_mathematics|5_2024-02-11T14-43-14.190988.parquet
        • **/details_harness|hendrycksTest-college_medicine|5_2024-02-11T14-43-14.190988.parquet
        • **/details_harness|hendrycksTest-college_physics|5_2024-02-11T14-43-14.190988.parquet
        • **/details_harness|hendrycksTest-computer_security|5_2024-02-11T14-43-14.190988.parquet
        • **/details_harness|hendrycksTest-conceptual_physics|5_2024-02-11T14-43-14.190988.parquet
        • **/details_harness|hendrycksTest-econometrics|5_2024-02-11T14-43-14.190988.parquet
        • **/details_harness|hendrycksTest-electrical_engineering|5_2024-02-11T14-43-14.190988.parquet
        • **/details_harness|hendrycksTest-elementary_mathematics|5_2024-02-11T14-43-14.190988.parquet
        • **/details_harness|hendrycksTest-formal_logic|5_2024-02-11T14-43-14.190988.parquet
        • **/details_harness|hendrycksTest-global_facts|5_2024-02-11T14-43-14.190988.parquet
        • **/details_harness|hendrycksTest-high_school_biology|5_2024-02-11T14-43-14.190988.parquet
        • ...

最新结果

以下是 2024-02-11T14:43:14.190988 运行的最新结果：

python { "all": { "acc": 0.5967272209929594, "acc_stderr": 0.03347226424165155, "acc_norm": 0.6019495737279535, "acc_norm_stderr": 0.034142369952769536, "mc1": 0.3880048959608323, "mc1_stderr": 0.01705876150134797, "mc2": 0.5289794791431763, "mc2_stderr": 0.015597946760218941 }, "harness|arc:challenge|25": { "acc": 0.575938566552901, "acc_stderr": 0.014441889627464398, "acc_norm": 0.6203071672354948, "acc_norm_stderr": 0.014182119866974872 }, "harness|hellaswag|10": { "acc": 0.6229834694284008, "acc_stderr": 0.004836486437527259, "acc_norm": 0.8194582752439753, "acc_norm_stderr": 0.003838519335886879 }, "harness|hendrycksTest-abstract_algebra|5": { "acc": 0.31, "acc_stderr": 0.046482319871173156, "acc_norm": 0.31, "acc_norm_stderr": 0.046482319871173156 }, "harness|hendrycksTest-anatomy|5": { "acc": 0.5777777777777777, "acc_stderr": 0.04266763404099582, "acc_norm": 0.5777777777777777, "acc_norm_stderr": 0.04266763404099582 }, "harness|hendrycksTest-astronomy|5": { "acc": 0.6052631578947368, "acc_stderr": 0.039777499346220734, "acc_norm": 0.6052631578947368, "acc_norm_stderr": 0.039777499346220734 }, "harness|hendrycksTest-business_ethics|5": { "acc": 0.54, "acc_stderr": 0.05009082659620332, "acc_norm": 0.54, "acc_norm_stderr": 0.05009082659620332 }, "harness|hendrycksTest-clinical_knowledge|5": { "acc": 0.6830188679245283, "acc_stderr": 0.028637235639800893, "acc_norm": 0.6830188679245283, "acc_norm_stderr": 0.028637235639800893 }, "harness|hendrycksTest-college_biology|5": { "acc": 0.6666666666666666, "acc_stderr": 0.03942082639927213, "acc_norm": 0.6666666666666666, "acc_norm_stderr": 0.03942082639927213 }, "harness|hendrycksTest-college_chemistry|5": { "acc": 0.39, "acc_stderr": 0.049020713000019756, "acc_norm": 0.39, "acc_norm_stderr": 0.049020713000019756 }, "harness|hendrycksTest-college_computer_science|5": { "acc": 0.51, "acc_stderr": 0.05024183937956911, "acc_norm": 0.51, "acc_norm_stderr": 0.05024183937956911 }, "harness|hendrycksTest-college_mathematics|5": { "acc": 0.36, "acc_stderr": 0.04824181513244218, "acc_norm": 0.36, "acc_norm_stderr": 0.04824181513244218 }, "harness|hendrycksTest-college_medicine|5": { "acc": 0.630057803468208, "acc_stderr": 0.036812296333943194, "acc_norm": 0.630057803468208, "acc_norm_stderr": 0.036812296333943194 }, "harness|hendrycksTest-college_physics|5": { "acc": 0.47058823529411764, "acc_stderr": 0.04966570903978529, "acc_norm": 0.47058823529411764, "acc_norm_stderr": 0.04966570903978529 }, "harness|hendrycksTest-computer_security|5": { "acc": 0.73, "acc_stderr": 0.0446196043338474, "acc_norm": 0.73, "acc_norm_stderr": 0.0446196043338474 }, "harness|hendrycksTest-conceptual_physics|5": { "acc": 0.502127659574468, "acc_stderr": 0.032685726586674915, "acc_norm": 0.502127659574468, "acc_norm_stderr": 0.032685726586674915 }, "harness|hendrycksTest-econometrics|5": { "acc": 0.43859649122807015, "acc_stderr": 0.04668000738510455, "acc_norm": 0.43859649122807015, "acc_norm_stderr": 0.04668000738510455 }, "harness|hendrycksTest-electrical_engineering|5": { "acc": 0.5379310344827586, "acc_stderr": 0.04154659671707548, "acc_norm": 0.5379310344827586, "acc_norm_stderr": 0.04154659671707548 }, "harness|hendrycksTest-elementary_mathematics|5": { "acc": 0.35185185185185186, "acc_stderr": 0.024594975128920945, "