JingweiNi/ocr2_hardest_questions_10k_seed20260527

--- dataset_info: features: - name: hardness_rank dtype: int64 - name: question dtype: string - name: answer dtype: string - name: question_id dtype: string - name: ocr2_id dtype: string - name: language_split dtype: string - name: source_dataset dtype: string - name: source_split dtype: string - name: source_platform dtype: string - name: difficulty dtype: string - name: difficulty_score dtype: float64 - name: difficulty_score_type dtype: string - name: judgement dtype: string - name: pass_rate dtype: string - name: source_index dtype: string - name: sample_seed dtype: int64 splits: - name: train num_bytes: 23260103 num_examples: 10000 download_size: 11173986 dataset_size: 23260103 configs: - config_name: default data_files: - split: train path: data/train-* --- # OCR2 Hardest Questions 10k This dataset contains the top 10,000 unique questions from `nvidia/OpenCodeReasoning-2` under a deterministic metadata ranking. In this dataset, "hardest" is defined only by OCR2 source metadata: difficulty first, then pass rate. It is not based on a verifier, model-generated traces, or annotation outcomes. ## Source - Source dataset: `nvidia/OpenCodeReasoning-2` - OCR2 language splits scanned: `train/python`, `train/cpp` - Sample seed: `20260527` - Local generation script: `synthetic_dataset_generation.prepare_ocr2_hardest_questions` ## Candidate Filtering The script scans OCR2 parquet metadata and constructs candidates as follows: 1. Keep rows with a non-empty `question_id`. 2. Keep only rows with `judgement == "right"`. 3. Convert `difficulty` to `difficulty_score`. 4. Drop unknown/unmapped difficulty values. 5. Collapse duplicate rows to one representative per `question_id`. The published run scanned `2,572,641` OCR2 rows and kept `33,706` unique candidate questions before taking the top 10,000. ## Difficulty Score Numeric difficulty values are treated as contest ratings and kept as-is. For example, OCR2 difficulty `3500` becomes `difficulty_score=3500.0`. String difficulty labels are mapped as follows: | Source style | Label | Score | |---|---:|---:| | TACO | `VERY_HARD` | 2500 | | TACO | `HARD` | 2200 | | TACO | `MEDIUM_HARD` | 1900 | | TACO | `MEDIUM` | 1500 | | TACO | `EASY` | 800 | | TACO | `UNKNOWN_DIFFICULTY` | -1 | | APPS | `competition` | 2200 | | APPS | `interview` | 1600 | | APPS | `introductory` | 800 | | Missing | `None` or empty string | -1 | Rows with score `-1` are removed because `--drop-unknown-difficulty` is enabled for this release. ## Duplicate Handling OCR2 can contain multiple rows for the same `question_id`, for example across languages or source solutions. The representative row for each `question_id` is chosen by: 1. Higher `difficulty_score`. 2. Lower `pass_rate`. 3. `judgement == "right"` if still tied. 4. Deterministic SHA-256 tie-break from the seed and row id. ## Final Hardness Ranking After deduplication, questions are sorted by: 1. `difficulty_score` descending. 2. `pass_rate` ascending. 3. Deterministic SHA-256 tie-break from the seed and `question_id`. 4. `question_id` as the final stable tie-break. The first 10,000 rows in this sorted list are published. The dataset column `hardness_rank` is this final zero-based rank. ## Question Text Reconstruction Many OCR2 rows store placeholder question text such as `-`. For the released dataset, question text is reconstructed from the original benchmark datasets using OCR2's `dataset`, `split`, and `index` metadata. The release validation found `0` placeholder questions and `0` missing questions after reconstruction. ## Release Statistics - Rows: `10,000` - Unique `question_id`: `10,000` - Judgement counts: `{'right': 10000}` - Source dataset counts: `apps=1356`, `code_contests=719`, `open-r1/codeforces=2894`, `taco=5031` - Difficulty score range: `1800.0` to `3500.0` - Difficulty counts: `1800=298`, `1900=347`, `2000=302`, `2100=306`, `2200=288`, `2300=249`, `2400=301`, `2500=262`, `2600=192`, `2700=202`, `2800=145`, `2900=135`, `3000=115`, `3100=108`, `3200=99`, `3300=74`, `3400=61`, `3500=129`, `HARD=1550`, `MEDIUM_HARD=1487`, `VERY_HARD=1994`, `competition=1356` ## Reproduction Command ```bash HF_HUB_DISABLE_XET=1 PYTHONPATH=. python -m synthetic_dataset_generation.prepare_ocr2_hardest_questions \ --sample-size 10000 \ --seed 20260527 \ --save-path gen_data/ocr2_hardest_questions_10k_seed20260527 \ --summary-path gen_data/ocr2_hardest_questions_10k_seed20260527_summary.json \ --hf-repo JingweiNi/ocr2_hardest_questions_10k_seed20260527 ``` The exact run summary is also stored in this repository as `summary.json`.This dataset contains the top 10,000 unique questions from `nvidia/OpenCodeReasoning-2` under a deterministic metadata ranking. In this dataset, hardest is defined only by OCR2 source metadata: difficulty first, then pass rate. It is not based on a verifier, model-generated traces, or annotation outcomes.

搜集汇总

数据集介绍

构建方式

该数据集衍生于NVIDIA发布的OpenCodeReasoning-2语料库，其构建遵循确定性元数据排序策略，从涵盖Python与CPP两种语言切分的二百五十七万余行原始记录中萃取高难度编程问题。构建流程先经候选过滤，保留具有非空question_id且判定为right的条目，并将异质难度标签映射为数值型难度分数，继而以question_id为键执行去重，代表行依难度降序、通过率升序及SHA-256确定性哈希次序竞逐，最终按难度分数与通过率联合排序取前一万条，且对占位符题目文本依据原始基准数据集元数据实施重构。

特点

数据集以万条唯一question_id构成，每例均带right判定，难度分数区间横跨一千八百至三千五百，涵盖APPS、CodeContests、Codeforces及TACO等多源平台，共含二十一种离散难度与来源标签，并附hardness_rank、pass_rate、language_split、source_dataset等十六维字段。题目文本经重构后无占位与缺失之虞，难度分布呈现长尾形态，既包含竞赛级高难评级，亦囊括TACO的VERY_HARD与HARD语义层级，凸显以元数据锚定之“最难”定义，而非依赖验证器或模型轨迹。

使用方法

使用该数据集可借助HuggingFace数据集库直接加载默认配置下的train切分，其下载体积约十一兆字节，内存展开约二十三兆字节，字段结构清晰便于筛选与排序。研究者可依据hardness_rank重排或分层抽样，利用difficulty_score与pass_rate构建课程学习或难度感知评测，亦可结合language_split与source_platform开展跨语言与跨来源的泛化分析。复现命令以PYTHONPATH与HF_HUB_DISABLE_XET环境变量配合模块化脚本，指定样本量、种子与保存路径即可重建相同子集，且仓库内summary.json提供精确运行摘要以供校验。

背景与挑战

背景概述

代码推理与程序合成领域长期依赖高难度竞赛题目作为评测基准，然而现有数据集多按来源或标签粗粒度组织，缺乏对题目内在难度的精细量化与严格去重。在此背景下，研究者从NVIDIA发布的OpenCodeReasoning-2（OCR2）中提取并构建了ocr2_hardest_questions_10k_seed20260527数据集，由JingweiNi等人于2026年发布。该数据集以难度评分与通过率为核心排序依据，从257万余条记录中筛选出10000道最具挑战性的唯一题目，涵盖APPS、TACO、CodeContests与Codeforces等来源，难度评分介于1800至3500之间。其核心贡献在于为代码推理模型提供了一套经过确定性元数据排序的困难评测集，推动了高难度代码生成与推理能力的边界测试。

当前挑战

该数据集所应对的领域问题在于：如何从海量异构代码题目中客观识别并排序出真正具有挑战性的问题，避免模型通过记忆或浅层模式匹配取得虚高表现。其具体挑战包括：一、题目元数据质量参差不齐，难度标签存在多种来源风格（TACO、APPS等），需统一映射为数值评分并剔除未知难度值；二、同一question_id可能对应多语言、多解法重复记录，需设计确定性去重策略以保留最具代表性的高难样本；三、OCR2原始数据中大量题目文本以占位符形式存储，须借助原始基准数据集的dataset、split与index元数据重建题干，确保问题完整可用；四、排序需兼顾难度评分降序与通过率升序，并引入SHA-256种子确定性打破平局，保证结果可复现且无偏。

常用场景

经典使用场景

在代码生成与程序推理的评测实践中，该数据集通常被用作衡量模型在极端困难编程任务上推理能力的基准。选取OpenCodeReasoning-2中难度评分最高且通过率最低的一万道题目，构成一个高难度的评估集，研究者借此检验大模型在复杂算法、数据结构以及竞赛级问题上的表现，尤其关注模型在长链条推理中保持逻辑一致性的能力。

衍生相关工作

自发布以来，该数据集衍生了若干围绕高难度代码推理的经典工作，包括针对OCR2元数据排序策略的改进研究、基于该数据集微调的代码大模型，以及利用其进行推理链验证与自我修正的方法。这些工作进一步拓展了代码推理评测的边界，并促进了难度感知训练与评测范式的形成。

数据集最近研究

最新研究方向

在代码大模型推理能力评估日益精细化的趋势下，该数据集聚焦于从OpenCodeReasoning-2中按难度与通过率元数据确定性筛选出的万道高难编程题，为竞赛级代码生成与推理基准提供了极端困难样本的策展范式。相关研究正围绕如何在不依赖验证器或模型生成轨迹的前提下，纯粹依据源元数据构建具备可复现性的挑战性评测集，以揭示当前先进模型在算法竞赛高阶问题上的真实能力边界。该方向紧密关联代码推理、难度分层评估以及基准污染控制等热点议题，对推动鲁棒且透明的代码智能评价体系建设具有重要的方法论意义。