hyunseoki/openthoughts3-dedup-index

---
license: odc-by
task_categories:
- text-generation
language:
- en
tags:
- reasoning
- math
- code
- science
- deduplication
- openthoughts
size_categories:
- 10K<n<100K
source_datasets:
- open-thoughts/OpenThoughts3-1.2M
pretty_name: OpenThoughts3 Dedup Index
---

# OpenThoughts3 Dedup Index

A deduplicated index over
[`open-thoughts/OpenThoughts3-1.2M`](https://huggingface.co/datasets/open-thoughts/OpenThoughts3-1.2M).
The upstream dataset contains ~18× duplicate problem statements (the same
question paired with many solver trajectories). This index keeps exactly one
canonical record per unique problem, making uniform random sampling of
distinct questions trivial.

## Summary

- **rows_scanned**: 1200000
- **unique_questions**: 65047
- **unique_with_gt_answer**: 45622
- **duplicate_ratio**: 18.45
- **domain_total_rows**:
- `code`: 250000
- `math`: 850000
- `science`: 100000
- **domain_unique_questions**:
- `code`: 5693
- `math`: 53105
- `science`: 6249
- **top_sources_by_unique**:
- `ai2-adapt-dev/openmath-2-math`: 53105
- `nvidia/OpenCodeReasoning`: 2007
- `organic-chemistry-questions`: 3743
- `stackexchange-physics`: 2506
- `stackexchange_codegolf`: 3686

## Schema

Each row of `openthoughts3_dedup.jsonl` has the following fields:

| Field | Type | Description |
|---|---|---|
| `hash` | str | md5 of normalized (whitespace-collapsed, lowercased) problem text |
| `problem` | str | The problem statement (the `human` turn of the upstream `conversations`) |
| `gt_answer` | str or null | `\boxed{...}` answer extracted from any matching upstream solver response (may be null for code-style problems without a boxed target) |
| `domain` | str | Upstream `domain` field: one of `math`, `code`, `science` |
| `source` | str | Upstream `source` field (e.g. `ai2-adapt-dev/openmath-2-math`, `stackexchange-physics`, `nvidia/OpenCodeReasoning`) |
| `difficulty` | str or null | Upstream `difficulty` value if present |
| `duplicate_count` | int | How many times this question appeared across the 1.2M source rows |
| `first_row_index` | int | Index within the upstream dataset of the first occurrence (for traceability) |

## Build

Produced by `scripts/build_openthoughts_dedup_index.py` in the
`memory_reasoning_split` research repo. The script streams the full
1.2M rows of the upstream dataset, MD5-hashes the normalized problem
text, keeps the first-seen record per hash, updates the cached
`gt_answer` if any later duplicate contained a boxed answer, and writes
one jsonl row per unique question plus a summary JSON.

## Intended use

Use this as the sampling pool when building self-distillation or
teacher-forcing reasoning datasets over OpenThoughts3 — uniform random
sampling on the raw 1.2M file is dominated by intra-cluster duplicates,
especially for the `code` split (44× duplicate ratio).

## License / Attribution

This index only stores problem statements and metadata derived from
OpenThoughts3. Please follow the upstream
[`open-thoughts/OpenThoughts3-1.2M`](https://huggingface.co/datasets/open-thoughts/OpenThoughts3-1.2M)
license terms.

许可证：odc-by
任务类别：
- 文本生成
语言：
- 英语
标签：
- 推理
- 数学
- 代码
- 科学
- 去重
- OpenThoughts
规模类别：
- 10K<n<100K
源数据集：
- open-thoughts/OpenThoughts3-1.2M
展示名称：OpenThoughts3 去重索引

# OpenThoughts3 去重索引

本数据集是针对[`open-thoughts/OpenThoughts3-1.2M`](https://huggingface.co/datasets/open-thoughts/OpenThoughts3-1.2M)构建的去重索引。其上游数据集包含约18倍的重复问题表述（即同一问题对应多条求解轨迹）。本索引为每个唯一问题保留一条规范记录，可轻松实现对不同问题的均匀随机采样。

## 摘要

- **扫描行数**：1,200,000
- **唯一问题数**：65,047
- **带有标准答案的唯一问题数**：45,622
- **重复率**：18.45
- **各领域总行数**：
- `code`：250,000
- `math`：850,000
- `science`：100,000
- **各领域唯一问题数**：
- `code`：5,693
- `math`：53,105
- `science`：6,249
- **按唯一问题数排名的Top数据源**：
- `ai2-adapt-dev/openmath-2-math`：53,105
- `nvidia/OpenCodeReasoning`：2,007
- `organic-chemistry-questions`：3,743
- `stackexchange-physics`：2,506
- `stackexchange_codegolf`：3,686

## 数据结构

`openthoughts3_dedup.jsonl` 的每一行包含以下字段：

| 字段名 | 数据类型 | 描述 |
|---|---|---|
| `hash` | 字符串 | 经归一化（空格折叠、小写转换）后的问题文本的MD5哈希值 |
| `problem` | 字符串 | 问题表述，对应上游数据集`conversations`中的`human`轮次内容 |
| `gt_answer` | 字符串或空值 | 从匹配的上游求解器响应中提取的`oxed{...}`格式答案；对于无框定目标的代码类问题，该字段可为空 |
| `domain` | 字符串 | 上游数据集的`domain`字段，取值为`math`（数学）、`code`（代码）或`science`（科学）之一 |
| `source` | 字符串 | 上游数据集的`source`字段，例如`ai2-adapt-dev/openmath-2-math`、`stackexchange-physics`、`nvidia/OpenCodeReasoning`等 |
| `difficulty` | 字符串或空值 | 上游数据集提供的`difficulty`字段，若存在则保留 |
| `duplicate_count` | 整数 | 该问题在120万条源数据行中出现的总次数 |
| `first_row_index` | 整数 | 该问题在原始上游数据集中首次出现的行索引，用于可追溯性 |

## 构建流程

本数据集由`memory_reasoning_split`研究仓库中的`scripts/build_openthoughts_dedup_index.py`脚本生成。该脚本流式处理上游数据集的全部120万行数据，对归一化后的问题文本计算MD5哈希，为每个哈希值保留首次出现的记录；若后续重复数据中包含框定格式的标准答案，则更新缓存的`gt_answer`字段，最终为每个唯一问题生成一条JSONL行，并附带摘要JSON文件。

## 预期用途

当基于OpenThoughts3构建自蒸馏（self-distillation）或教师强制（teacher-forcing）推理数据集时，可将本数据集用作采样池。直接对原始120万条数据文件进行均匀随机采样会受集群内重复数据的主导，尤其针对`code`（代码）领域子集，其重复率达44倍。

## 许可证与归因

本索引仅存储从OpenThoughts3衍生的问题表述与元数据，请遵守上游数据集[`open-thoughts/OpenThoughts3-1.2M`](https://huggingface.co/datasets/open-thoughts/OpenThoughts3-1.2M)的许可证条款。