microsoft/OpenMementos

--- language: - en license: mit size_categories: - 100K<n<1M task_categories: - text-generation tags: - reasoning - chain-of-thought - context-compression - synthetic - memento pretty_name: OpenMementos-228K dataset_info: - config_name: default features: - name: problem dtype: string - name: response dtype: string - name: domain dtype: string - name: source dtype: string - name: difficulty dtype: int64 splits: - name: train num_examples: 228557 - config_name: full features: - name: problem dtype: string - name: response dtype: string - name: domain dtype: string - name: source dtype: string - name: difficulty dtype: int64 - name: sentences sequence: string - name: blocks sequence: sequence: int64 - name: block_summaries sequence: string splits: - name: train num_examples: 228557 configs: - config_name: default data_files: - split: train path: data/train-* default: true - config_name: full data_files: - split: train path: full/train-* --- # OpenMementos-228K A dataset of **228,557** reasoning traces annotated with block segmentation and compressed summaries (mementos), derived from [OpenThoughts-v3](https://huggingface.co/datasets/open-thoughts/OpenThoughts3-1.2M). Memento is a framework for teaching language models to **manage their own context** during long-form reasoning. Instead of generating one long, unstructured chain-of-thought, memento-trained models segment their reasoning into **blocks**, compress each block into a dense **summary** (a *memento*), and continue reasoning from mementos alone. In the released training data, the paper reports **~6× trace-level compression**, from ~10,900 block tokens to ~1,850 memento tokens per trace. Code: [microsoft/memento](https://github.com/microsoft/memento) ## Quick Start ```python from datasets import load_dataset # Training-ready format (default) ds = load_dataset("microsoft/OpenMementos", split="train") # With pipeline components (sentences, blocks, summaries) ds = load_dataset("microsoft/OpenMementos", "full", split="train") ``` ## Available Subsets ### `default` — Training-ready data Contains pre-assembled memento-formatted responses ready for SFT training. | Column | Type | Description | |--------|------|-------------| | `problem` | string | Problem statement | | `response` | string | Memento-formatted response with block/summary tokens (see format below) | | `domain` | string | `code`, `math`, or `science` | | `source` | string | Original dataset source | | `difficulty` | int | Difficulty rating (code domain only, 6–10) | ### `full` — With pipeline components Everything in `default`, plus the intermediate pipeline outputs for researchers who want to re-segment, analyze, or reconstruct: | Column | Type | Description | |--------|------|-------------| | `sentences` | list[string] | Individual sentences from sentence splitting | | `blocks` | list[list[int]] | Block boundaries as `[start_idx, end_idx]` sentence index pairs | | `block_summaries` | list[string] | Iteratively refined summary for each block | ## Response Format The `response` column contains reasoning traces with Memento special tokens: ``` <think> <\|block_start\|> [reasoning sentences for block 1] <\|block_end\|> <\|summary_start\|> [compressed summary of block 1] <\|summary_end\|> <\|block_start\|> [reasoning sentences for block 2] <\|block_end\|> <\|summary_start\|> [compressed summary of block 2] <\|summary_end\|> ... </think> [final answer] ``` ### Special Tokens | Token | Purpose | |-------|---------| | `<think>` / `</think>` | Reasoning wrapper | | `<\|block_start\|>` / `<\|block_end\|>` | Reasoning block delimiters | | `<\|summary_start\|>` / `<\|summary_end\|>` | Summary (memento) delimiters | During inference with block masking, completed blocks are evicted from the KV cache and the model continues reasoning from the summary tokens alone. ## Dataset Statistics ### Overview | Statistic | Value | |-----------|-------| | Total examples | 228,557 | | Math examples | 123,333 (54%) | | Science examples | 61,485 (27%) | | Code examples | 43,739 (19%) | | Avg sentences per example | 187 | ### Block & Summary Statistics (paper-aligned) | Statistic | Value | |-----------|-------| | Median blocks per example | Math: ~9, Code: ~9, Science: ~7 | | Median block size | Ranges from ~2.3K chars (science) to ~3.8K chars (math) | | Median summary size | ~509–603 chars across domains | | Median compression ratio | Math: 0.16, Code: 0.18, Science: 0.23 | | Block-level compression | ~4×–6× depending on domain | | Average block length | ~1,150 tokens | | Average memento length | ~194 tokens | | Trace-level compression | ~6× (from ~10,900 block tokens to ~1,850 memento tokens per trace) | ### Summary Quality | Statistic | Value | |-----------|-------| | Single-pass pass rate (≥ 8/10) | 28% | | After iterative refinement (≥ 8/10) | 92% | | Judge threshold | 8/10 | | Max judge-feedback rounds used to build the dataset | 2 | ## Data Sources All reasoning traces are derived from [OpenThoughts-v3](https://huggingface.co/datasets/open-thoughts/OpenThoughts3-1.2M). The released 228K traces inherit the following source composition: | Source | Domain | Count | |--------|--------|-------| | [ai2-adapt-dev/openmath-2-math](https://huggingface.co/datasets/ai2-adapt-dev/openmath-2-math) | math | 123,333 | | [organic-chemistry-questions](https://huggingface.co/datasets/organic-chemistry-questions) | science | 39,097 | | [nvidia/OpenCodeReasoning](https://huggingface.co/datasets/nvidia/OpenCodeReasoning) | code | 25,155 | | [stackexchange-physics](https://huggingface.co/datasets/stackexchange-physics) | science | 22,388 | | [stackexchange_codegolf](https://huggingface.co/datasets/stackexchange_codegolf) | code | 18,584 | ## Data Pipeline The dataset was constructed through the following pipeline: 1. **Sentence Splitting** — Original chain-of-thought traces are partitioned into atomic reasoning sentences while preserving code and multi-line math; this reduces candidate boundaries from ~397 to ~187 per trace on average. 2. **Boundary Scoring** — An LLM scores each sentence boundary on a **0–3** scale for how suitable it is as a block break point. 3. **Block Segmentation** — Sentences are grouped into blocks with algorithmic optimization over boundary scores, while enforcing a minimum block size of 200 tokens and penalizing highly unbalanced partitions. 4. **Summary Generation** — Each block is compressed into a memento that preserves the logically relevant reasoning state needed for subsequent blocks, targeting ~15–25% of the original tokens. 5. **Iterative Refinement** — Summaries are evaluated by an LLM judge on a 0–10 rubric and refined with judge feedback for up to two rounds until they meet a quality threshold (≥8/10). ## Training This dataset can be used to fine-tune models for memento-style generation. The typical training setup: ```python # Example: SFT with the default subset from datasets import load_dataset ds = load_dataset("microsoft/OpenMementos", split="train") # Each example has: problem (user message), response (assistant message) # Format as chat messages for your trainer: def format_chat(example): return { "messages": [ {"role": "user", "content": example["problem"]}, {"role": "assistant", "content": example["response"]}, ] } ds = ds.map(format_chat) ``` ## License MIT License. See [LICENSE](LICENSE) for details. ## Citation ```bibtex @article{memento2026, author={Vasilis Kontonis and Yuchen Zeng and Shivam Garg and Lingjiao Chen and Hao Tang and Ziyan Wang and Ahmed Awadallah and Eric Horvitz and John Langford and Dimitris Papailiopoulos}, title={Memento: Teaching LLMs to Manage Their Own Context}, year={2026}, } ```--- 语言： - 英语 许可证： - MIT协议 规模类别： - 10万 < 样本数 < 100万 任务类别： - 文本生成 标签： - 推理 - 思维链（chain-of-thought） - 上下文压缩（context-compression） - 合成数据 - memento 美观名称：OpenMementos-228K 数据集信息： - 配置名称：default 特征： - 名称：problem 数据类型：字符串 - 名称：response 数据类型：字符串 - 名称：domain 数据类型：字符串 - 名称：source 数据类型：字符串 - 名称：difficulty 数据类型：64位整数 划分： - 名称：train 样本数：228557 - 配置名称：full 特征： - 名称：problem 数据类型：字符串 - 名称：response 数据类型：字符串 - 名称：domain 数据类型：字符串 - 名称：source 数据类型：字符串 - 名称：difficulty 数据类型：64位整数 - 名称：sentences 数据类型：字符串序列 - 名称：blocks 数据类型：整数序列的序列 - 名称：block_summaries 数据类型：字符串序列 划分： - 名称：train 样本数：228557 配置项： - 配置名称：default 数据文件： - 划分：train 路径：data/train-* 默认启用：true - 配置名称：full 数据文件： - 划分：train 路径：full/train-* --- # OpenMementos-228K 本数据集包含**228,557**条推理轨迹，标注了块分割与压缩摘要（memento），源自[OpenThoughts-v3](https://huggingface.co/datasets/open-thoughts/OpenThoughts3-1.2M)。 memento是一种用于教授大语言模型（LLM）在长文本推理过程中**自主管理上下文**的框架。与生成单一冗长且无结构的思维链不同，采用memento训练的模型会将推理过程划分为**块（block）**，并将每个块压缩为一段紧凑的**摘要（即memento）**，后续仅基于该摘要继续推理。在本次发布的训练数据中，论文报告称**轨迹级压缩比约为6倍**，单条轨迹的块令牌（Token）数约为10900，压缩后仅需约1850个memento令牌。 代码仓库：[microsoft/memento](https://github.com/microsoft/memento) ## 快速上手 python from datasets import load_dataset # 适配训练的格式（默认配置） ds = load_dataset("microsoft/OpenMementos", split="train") # 包含流水线组件（分句、块、摘要） ds = load_dataset("microsoft/OpenMementos", "full", split="train") ## 可用子集 ### `default` — 适配训练的数据集 包含预组装的memento格式响应，可直接用于监督微调（SFT）训练。 | 列名 | 类型 | 说明 | |--------|------|-------------| | `problem` | 字符串 | 问题陈述 | | `response` | 字符串 | 采用memento格式的响应，包含块/摘要令牌（详见下文格式） | | `domain` | 字符串 | 可选值：`code`（代码）、`math`（数学）或`science`（科学） | | `source` | 字符串 | 原始数据集来源 | | `difficulty` | 64位整数 | 难度评分（仅代码域有效，取值范围6~10） | ### `full` — 包含流水线组件的数据集 包含`default`子集的全部内容，额外提供中间流水线输出，供研究人员重新分割、分析或重构数据使用： | 列名 | 类型 | 说明 | |--------|------|-------------| | `sentences` | 字符串列表 | 分句拆分后的独立句子 | | `blocks` | 整数列表的列表 | 块边界，以`[起始句索引, 结束句索引]`的句索引对形式表示 | | `block_summaries` | 字符串列表 | 每个块经过迭代优化后的摘要 | ## 响应格式 `response`列包含带有memento特殊令牌的推理轨迹： <think> <|block_start|> [块1的推理句子] <|block_end|> <|summary_start|> [块1的压缩摘要] <|summary_end|> <|block_start|> [块2的推理句子] <|block_end|> <|summary_start|> [块2的压缩摘要] <|summary_end|> ... </think> [最终答案] ### 特殊令牌 | 令牌 | 用途 | |-------|---------| | `<think>` / `</think>` | 推理内容包装符 | | `<|block_start|>` / `<|block_end|>` | 推理块分隔符 | | `<|summary_start|>` / `<|summary_end|>` | 摘要（memento）分隔符 | 在使用块掩码进行推理时，已完成的块会从KV缓存中移除，模型仅基于摘要令牌继续推理。 ## 数据集统计信息 ### 概览 | 统计项 | 数值 | |-----------|-------| | 总样本数 | 228,557 | | 数学域样本数 | 123,333（占比54%） | | 科学域样本数 | 61,485（占比27%） | | 代码域样本数 | 43,739（占比19%） | | 单样本平均分句数 | 187 | ### 块与摘要统计信息（与论文对齐） | 统计项 | 数值 | |-----------|-------| | 单样本块数中位数 | 数学域约9个，代码域约9个，科学域约7个 | | 块大小中位数 | 范围约2300字符（科学域）~3800字符（数学域） | | 摘要大小中位数 | 各域约509~603字符 | | 压缩比中位数 | 数学域0.16，代码域0.18，科学域0.23 | | 块级压缩比 | 依域不同约4~6倍 | | 单块平均长度 | 约1150个Token | | 单memento平均长度 | 约194个Token | | 轨迹级压缩比 | 约6倍（单轨迹块令牌数约10900，压缩后仅需约1850个memento令牌） | ### 摘要质量 | 统计项 | 数值 | |-----------|-------| | 单轮生成通过率（≥8/10） | 28% | | 经迭代优化后通过率（≥8/10） | 92% | | 评判阈值 | 8/10 | | 构建数据集时使用的最大评判反馈轮次 | 2轮 | ## 数据来源 所有推理轨迹均源自[OpenThoughts-v3](https://huggingface.co/datasets/open-thoughts/OpenThoughts3-1.2M)。本次发布的22.8万条轨迹继承了以下来源构成： | 来源 | 领域 | 样本数 | |--------|--------|-------| | [ai2-adapt-dev/openmath-2-math](https://huggingface.co/datasets/ai2-adapt-dev/openmath-2-math) | 数学 | 123,333 | | [organic-chemistry-questions](https://huggingface.co/datasets/organic-chemistry-questions) | 科学 | 39,097 | | [nvidia/OpenCodeReasoning](https://huggingface.co/datasets/nvidia/OpenCodeReasoning) | 代码 | 25,155 | | [stackexchange-physics](https://huggingface.co/datasets/stackexchange-physics) | 科学 | 22,388 | | [stackexchange_codegolf](https://huggingface.co/datasets/stackexchange_codegolf) | 代码 | 18,584 | ## 数据流水线 本数据集通过以下流水线构建： 1. **分句拆分** — 将原始思维链轨迹拆分为独立的推理句子，同时保留代码与多行数学表达式，平均每条轨迹的候选边界从约397个减少至约187个。 2. **边界评分** — 使用大语言模型对每个句子边界进行0~3分的评分，衡量其作为块分割点的适配程度。 3. **块分割** — 通过对边界评分进行算法优化，将句子分组为块，同时强制要求块的最小令牌数为200，并惩罚极度不均衡的分区。 4. **摘要生成** — 将每个块压缩为一段memento摘要，保留后续推理所需的逻辑相关推理状态，目标压缩比例为原令牌数的15%~25%。 5. **迭代优化** — 由大语言模型评判员基于0~10分的评分标准对摘要进行评估，并根据评判反馈最多进行两轮优化，直至摘要达到质量阈值（≥8/10）。 ## 训练 本数据集可用于微调支持memento风格生成的模型。典型训练流程如下： python # 示例：使用默认子集进行监督微调（SFT） from datasets import load_dataset ds = load_dataset("microsoft/OpenMementos", split="train") # 每个样本包含：problem（用户提问）、response（助手回复） # 按聊天消息格式格式化，适配你的训练器： def format_chat(example): return { "messages": [ {"role": "user", "content": example["problem"]}, {"role": "assistant", "content": example["response"]}, ] } ds = ds.map(format_chat) ## 许可证 MIT许可证。详见[LICENSE](LICENSE)文件。 ## 引用 bibtex @article{memento2026, author={Vasilis Kontonis and Yuchen Zeng and Shivam Garg and Lingjiao Chen and Hao Tang and Ziyan Wang and Ahmed Awadallah and Eric Horvitz and John Langford and Dimitris Papailiopoulos}, title={Memento: Teaching LLMs to Manage Their Own Context}, year={2026}, }