sam-paech/magi_irt_1_0

---
license: mit
dataset_info:
features:
- name: query
dtype: string
- name: choices
sequence: string
- name: gold
sequence: int64
- name: source
dtype: string
splits:
- name: test
num_bytes: 1578275
num_examples: 2154
download_size: 802713
dataset_size: 1578275
configs:
- config_name: default
data_files:
- split: test
path: data/test-*
---

## 🧙MAGI: A hard subset of MMLU and AGIEval✨

[Click for the long version](https://sampaech.substack.com/p/creating-magi-a-hard-subset-of-mmlu).

LLM Benchmarks are chasing a moving target and fast running out of headroom. They are struggling to effectively separate SOTA models from leaderboard optimisers. Can we salvage these old dinosaurs for scrap and make a better benchmark?

I created two subsets of MMLU + AGIEval:

MAGI-Hard: 3203 questions, 4x more discriminative between top models (as measured by std. dev.) This subset is brutal to 7b models and useful for exposing differences between high ability models. Downside: a reference model (Deepseek-67b) is “burned” and cannot be scored fairly by this subset.

MAGI-IRT: 2154 questions, 2x more discriminative. This subset is more balanced and retains discriminative power for low + mid ability models. It uses Item Response Theory (IRT) to model question difficulty, and can score all models fairly.

You can find the MAGI subsets [here](https://huggingface.co/sam-paech) and use them with [this fork of the Eleuther eval harness](https://github.com/sqrkl/lm-evaluation-harness).

MAGI has been added as a metric on the [EQ-Bench leaderboard](https://eqbench.com/).

原始信息汇总

数据集概述

数据集信息

• 许可证: MIT
• 数据集大小: 1578275字节
• 下载大小: 802713字节
• 示例数量: 2154

数据集特征

• query: 字符串类型
• choices: 字符串序列
• gold: 整数序列（int64）
• source: 字符串类型

数据集划分

• 测试集:
  • 大小: 1578275字节
  • 示例数量: 2154

配置

• 默认配置:
  • 数据文件路径: data/test-*