trillionlabs/rBridge

--- license: apache-2.0 task_categories: - text-generation language: - en tags: - reasoning - logprobs - token-probability - rbridge - proxy-model - scaling-laws pretty_name: "rBridge Paper's Reasoning Traces & Token Logprobs" size_categories: - 1K<n<10K configs: - config_name: arc_challenge data_files: - split: test path: traces/arc_challenge/gpt4o_s1_new.json - config_name: cqa data_files: - split: test path: traces/cqa/gpt4o_s1.json - config_name: gsm8k data_files: - split: test path: traces/gsm8k/gpt4o_s1_new.json - config_name: humaneval data_files: - split: test path: traces/humaneval/gpt4o_s1.json - config_name: math500 data_files: - split: test path: traces/math500/gpt4o_s1_merged.json - config_name: mmlu_pro data_files: - split: test path: traces/mmlu_pro/gpt4o_s1_stem_2_new.json --- # 🌉 rBridge Paper's Reasoning Traces & Token Logprobs This dataset contains GPT-4o reasoning traces and token-level logprobs for six reasoning benchmarks, released as part of the [rBridge](https://github.com/trillion-labs/rBridge) project ([paper](https://arxiv.org/abs/2509.21013)). rBridge uses these traces as gold-label reasoning references. By computing a **weighted negative log-likelihood** over these traces — where each token is weighted by the frontier model's confidence — small proxy models (≤1B) can reliably predict the reasoning performance of much larger LLMs (7B–32B+). ## 📊 Benchmarks | Config | Benchmark | Traces | Logprob Tokens | Logprob Files | |---|---|---:|---:|---:| | `arc_challenge` | ARC-Challenge | 1,172 | 143,492 | 3 parts | | `cqa` | CommonsenseQA | 1,221 | 149,350 | 3 parts | | `gsm8k` | GSM8K | 1,319 | 178,309 | 3 parts | | `humaneval` | HumanEval | 164 | 42,290 | 1 file | | `math500` | MATH-500 | 495 | 174,636 | 3 parts | | `mmlu_pro` | MMLU-Pro (STEM) | 5,791 | 1,627,990 | 31 parts | | | **Total** | **10,162** | **2,316,067** | | ## 📁 File Structure Each benchmark directory under `traces/` contains: 1. **Reasoning traces** (`.json`) — A JSON array of GPT-4o completions with reasoning and final answers. 2. **Token logprobs** (`.jsonl`) — One row per token from the GPT-4o completion, with full top-k logprobs. Split into multiple parts for large benchmarks. ``` traces/ ├── arc_challenge/ │ ├── gpt4o_s1_new.json # reasoning traces │ ├── gpt4o_s1_new_logprobs_part01_of_03.jsonl # token logprobs │ ├── gpt4o_s1_new_logprobs_part02_of_03.jsonl │ └── gpt4o_s1_new_logprobs_part03_of_03.jsonl ├── cqa/ │ ├── gpt4o_s1.json │ └── gpt4o_s1_logprobs_part{01..03}_of_03.jsonl ├── gsm8k/ │ ├── gpt4o_s1_new.json │ └── gpt4o_s1_new_logprobs_part{01..03}_of_03.jsonl ├── humaneval/ │ ├── gpt4o_s1.json │ └── gpt4o_s1_logprobs.jsonl ├── math500/ │ ├── gpt4o_s1_merged.json │ └── gpt4o_s1_logprobs_part{01..03}_of_03_merged.jsonl └── mmlu_pro/ ├── gpt4o_s1_stem_2_new.json └── gpt4o_s1_stem_2_new_logprobs_part{01..31}_of_31.jsonl ``` ## 🔖 Schema ### Reasoning Traces (`.json`) Each entry in the JSON array contains: | Field | Type | Description | |---|---|---| | `doc_id` | int | Document index | | `sample_id` | int | Sample index | | `dataset` | string | Benchmark name | | `original_question` | string | Input question / prompt | | `expected_answer` | string | Reference answer text | | `ground_truth_final_answer` | string | Ground-truth label (e.g., `"C"`) | | `gpt4o_reasoning` | string | GPT-4o chain-of-thought reasoning | | `gpt4o_final_answer` | string | GPT-4o predicted answer | | `model` | string | Model identifier (`openai/gpt-4o`) | | `usage` | object | Token usage (prompt, completion, total) | | `subject` | string | Subject / category (where applicable) | | `level` | string | Difficulty level (where applicable) | Additional benchmark-specific fields (e.g., `choices`, `task_id`, `question_id`) vary by dataset. ### Token Logprobs (`.jsonl`) Each line represents one token from the GPT-4o completion: | Field | Type | Description | |---|---|---| | `doc_id` | int | Document index (links to trace entry) | | `sample_id` | int | Sample index (links to trace entry) | | `position` | int | Token position in the completion | | `token` | string | The token string | | `logprob` | float | Log-probability assigned by GPT-4o | | `prob` | float | Probability (exp of logprob) | | `top_logprobs` | list | Top-k alternative tokens with their logprobs and probs | ## 🚀 Usage ### Load reasoning traces by benchmark ```python from datasets import load_dataset # Load a specific benchmark ds = load_dataset("trillionlabs/rBridge", "arc_challenge", split="test") print(ds[0]["gpt4o_reasoning"]) # Load another benchmark ds = load_dataset("trillionlabs/rBridge", "mmlu_pro", split="test") ``` ### Download logprobs files ```python from huggingface_hub import hf_hub_download # Download a single logprobs file path = hf_hub_download( repo_id="trillionlabs/rBridge", filename="traces/arc_challenge/gpt4o_s1_new_logprobs_part01_of_03.jsonl", repo_type="dataset", ) # Read it import json with open(path) as f: for line in f: token_data = json.loads(line) print(token_data["token"], token_data["prob"]) break ``` ### Download everything ```python from huggingface_hub import snapshot_download snapshot_download(repo_id="trillionlabs/rBridge", repo_type="dataset") ``` ## 🔗 Related Resources - **Paper**: [Predicting LLM Reasoning Performance with Small Proxy Model](https://arxiv.org/abs/2509.21013) ## 📝 Citation ```bibtex @inproceedings{ koh2026predicting, title={Predicting {LLM} Reasoning Performance with Small Proxy Model}, author={Woosung Koh and Juyoung Suk and Sungjun Han and Se-Young Yun and Jay Shin}, booktitle={The Fourteenth International Conference on Learning Representations}, year={2026}, url={https://openreview.net/forum?id=JSE40ljyKm} } ```许可证: apache-2.0 任务类别: - 文本生成 语言: - 英语 标签: - 推理 - 对数概率（logprobs） - Token概率（token-probability） - rBridge - 代理模型（proxy-model） - 缩放定律（scaling-laws） 漂亮名称: "rBridge论文的推理轨迹与Token对数概率" 规模类别: - 1K<n<10K 配置项: - 配置名称: arc_challenge 数据文件: - 拆分: 测试集 路径: traces/arc_challenge/gpt4o_s1_new.json - 配置名称: cqa 数据文件: - 拆分: 测试集 路径: traces/cqa/gpt4o_s1.json - 配置名称: gsm8k 数据文件: - 拆分: 测试集 路径: traces/gsm8k/gpt4o_s1_new.json - 配置名称: humaneval 数据文件: - 拆分: 测试集 路径: traces/humaneval/gpt4o_s1.json - 配置名称: math500 数据文件: - 拆分: 测试集 路径: traces/math500/gpt4o_s1_merged.json - 配置名称: mmlu_pro 数据文件: - 拆分: 测试集 路径: traces/mmlu_pro/gpt4o_s1_stem_2_new.json # 🌉 rBridge论文的推理轨迹与Token对数概率 该数据集包含6个推理基准测试的GPT-4o推理轨迹与Token级对数概率（logprobs），作为[rBridge](https://github.com/trillion-labs/rBridge)项目（[论文](https://arxiv.org/abs/2509.21013)）的附属资源发布。 rBridge将这些轨迹作为金标推理参考。通过在这些轨迹上计算**加权负对数似然**——其中每个Token的权重由前沿模型的置信度决定——参数规模不超过10亿的小型代理模型（proxy-model）可以可靠地预测规模大得多的大语言模型（Large Language Model，LLM）的推理性能。 ## 📊 基准测试 | 配置名称 | 基准测试 | 推理轨迹数 | Token对数概率数 | 日志概率文件数 | |---|---|---:|---:|---:| | `arc_challenge` | ARC-Challenge | 1,172 | 143,492 | 3个分块 | | `cqa` | 常识问答（CommonsenseQA） | 1,221 | 149,350 | 3个分块 | | `gsm8k` | GSM8K | 1,319 | 178,309 | 3个分块 | | `humaneval` | HumanEval | 164 | 42,290 | 1个文件 | | `math500` | MATH-500 | 495 | 174,636 | 3个分块 | | `mmlu_pro` | MMLU-Pro（STEM领域） | 5,791 | 1,627,990 | 31个分块 | | | **总计** | **10,162** | **2,316,067** | | ## 📁 文件结构 `traces/`目录下的每个基准测试子目录包含： 1. **推理轨迹**（`.json`）：包含GPT-4o生成结果的JSON数组，内含推理过程与最终答案。 2. **Token对数概率**（`.jsonl`）：每行对应GPT-4o生成结果中的一个Token，包含完整的Top-K对数概率。对于大规模基准测试，文件会拆分为多个分块。 traces/ ├── arc_challenge/ │ ├── gpt4o_s1_new.json # 推理轨迹文件 │ ├── gpt4o_s1_new_logprobs_part01_of_03.jsonl # Token对数概率分块1/3 │ ├── gpt4o_s1_new_logprobs_part02_of_03.jsonl # Token对数概率分块2/3 │ └── gpt4o_s1_new_logprobs_part03_of_03.jsonl # Token对数概率分块3/3 ├── cqa/ │ ├── gpt4o_s1.json │ └── gpt4o_s1_logprobs_part{01..03}_of_03.jsonl ├── gsm8k/ │ ├── gpt4o_s1_new.json │ └── gpt4o_s1_new_logprobs_part{01..03}_of_03.jsonl ├── humaneval/ │ ├── gpt4o_s1.json │ └── gpt4o_s1_logprobs.jsonl ├── math500/ │ ├── gpt4o_s1_merged.json │ └── gpt4o_s1_logprobs_part{01..03}_of_03_merged.jsonl └── mmlu_pro/ ├── gpt4o_s1_stem_2_new.json └── gpt4o_s1_stem_2_new_logprobs_part{01..31}_of_31.jsonl ## 🔖 模式 ### 推理轨迹（`.json`） JSON数组中的每个条目包含以下字段： | 字段名 | 数据类型 | 描述 | |---|---|---| | `doc_id` | 整数 | 文档索引 | | `sample_id` | 整数 | 样本索引 | | `dataset` | 字符串 | 基准测试名称 | | `original_question` | 字符串 | 输入问题/提示词 | | `expected_answer` | 字符串 | 参考答案文本 | | `ground_truth_final_answer` | 字符串 | 真实标注标签（例如：`"C"`） | | `gpt4o_reasoning` | 字符串 | GPT-4o的思维链推理过程 | | `gpt4o_final_answer` | 字符串 | GPT-4o预测的答案 | | `model` | 字符串 | 模型标识符（`openai/gpt-4o`） | | `usage` | 对象 | Token使用情况（提示Token、补全Token、总Token） | | `subject` | 字符串 | 主题/类别（如适用） | | `level` | 字符串 | 难度等级（如适用） | 此外，部分基准测试会包含特定于数据集的额外字段（例如`choices`、`task_id`、`question_id`）。 ### Token对数概率（`.jsonl`） 每行对应GPT-4o生成结果中的一个Token： | 字段名 | 数据类型 | 描述 | |---|---|---| | `doc_id` | 整数 | 文档索引（与轨迹条目关联） | | `sample_id` | 整数 | 样本索引（与轨迹条目关联） | | `position` | 整数 | 补全结果中的Token位置 | | `token` | 字符串 | Token字符串 | | `logprob` | 浮点数 | GPT-4o赋予该Token的对数概率 | | `prob` | 浮点数 | 概率值（对数概率的指数运算结果） | | `top_logprobs` | 列表 | Top-K替代Token及其对数概率与概率值 | ## 🚀 使用方法 ### 按基准测试加载推理轨迹 python from datasets import load_dataset # 加载指定基准测试 ds = load_dataset("trillionlabs/rBridge", "arc_challenge", split="test") print(ds[0]["gpt4o_reasoning"]) # 加载其他基准测试 ds = load_dataset("trillionlabs/rBridge", "mmlu_pro", split="test") ### 下载对数概率文件 python from huggingface_hub import hf_hub_download # 下载单个对数概率文件 path = hf_hub_download( repo_id="trillionlabs/rBridge", filename="traces/arc_challenge/gpt4o_s1_new_logprobs_part01_of_03.jsonl", repo_type="dataset", ) # 读取文件 import json with open(path) as f: for line in f: token_data = json.loads(line) print(token_data["token"], token_data["prob"]) break ### 下载全部数据集 python from huggingface_hub import snapshot_download snapshot_download(repo_id="trillionlabs/rBridge", repo_type="dataset") ## 🔗 相关资源 - **论文**：[使用小型代理模型预测大语言模型推理性能](https://arxiv.org/abs/2509.21013) ## 📝 引用 bibtex @inproceedings{ koh2026predicting, title={Predicting {LLM} Reasoning Performance with Small Proxy Model}, author={Woosung Koh and Juyoung Suk and Sungjun Han and Se-Young Yun and Jay Shin}, booktitle={The Fourteenth International Conference on Learning Representations}, year={2026}, url={https://openreview.net/forum?id=JSE40ljyKm} }