nvidia/nemotron-research-lgt

--- license: cc-by-nc-4.0 language: - en tags: - reasoning - vision - multimodal - vlm - chain-of-thought - nvidia size_categories: - 1M<n<10M configs: - config_name: sft_stage1 data_files: - split: train path: data/sft_stage1/train-*.parquet - config_name: sft_stage2 data_files: - split: train path: data/sft_stage2/train-*.parquet - config_name: sft_stage2_merged data_files: - split: train path: data/sft_stage2_merged/train-*.parquet - config_name: dpo_stage1 data_files: - split: train path: data/dpo_stage1/train-*.parquet - config_name: dpo_stage2 data_files: - split: train path: data/dpo_stage2/train-*.parquet --- <div align="center"> # LGT-1M+ **Long Grounded Thoughts: Synthesizing Visual Problems and Reasoning Chains at Scale** [](https://arxiv.org/abs/2511.05705) [](https://creativecommons.org/licenses/by-nc/4.0/) </div> **LGT** is a synthetic visual reasoning dataset containing **1M+ new visual reasoning problems** of varying complexity with reasoning traces exhibiting complex cognitive behaviours. The dataset supports the entire VLM post-training spectrum: SFT, offline RL (DPO), and online RL (GRPO/RLVR). This dataset is for research and development only. ## Long Grounded Thoughts (LGT) We identify two critical bottlenecks in current vision-centric reasoning data synthesis frameworks when pushed to scale: 1) **problem synthesis saturation** — relying solely on captions for problem synthesis limits the diversity and grounding of the generated vision-centric questions, causing downstream performance to saturate quickly, and 2) **cognitive simplicity** — the synthesized reasoning traces still lack higher-order reasoning structures, in part because of the simplicity of the underlying problems and lack of diversity in the synthesis. LGT tackles these core challenges focusing on **scale**, **complexity**, and **CoT richness**: - **Scale.** LGT overcomes the bottlenecks of caption-only synthesis methods by leveraging grounded object metadata, scaling novel problem synthesis from 30K to over 1M+ high-quality problems without performance saturation. - **Complexity.** LGT significantly increases vision-centric problem synthesis difficulty. The composition hardening algorithm reduces the rate of trivially solvable questions by ~10x (decreasing from 36.7% in baselines to 3.3%). - **CoT Richness.** LGT enhances the cognitive depth of the synthesized CoT traces. The method increases the frequency of complex cognitive behaviors by +206%, yielding traces that are 3x richer than the baseline. ## Dataset Overview | Config | Split | Examples | Format | Description | |--------|-------|----------|--------|-------------| | `sft_stage1` | train | 753,675 | SFT | Grounded MCQs with long CoT traces (~750K) | | `sft_stage2` | train | 163,873 | SFT | Composition-hardened MCQs with long CoT traces | | `sft_stage2_merged` | train | 243,873 | SFT | Stage 2 Composition-hardened MCQs with long CoT traces merged with additional data | | `dpo_stage1` | train | 129,318 | DPO | Preference pairs from Stage 1 MCQs | | `dpo_stage2` | train | 15,685 | DPO | Preference pairs from Stage 2 composition-hardened MCQs | **Source images:** [DOCCI](https://google.github.io/docci/) (Descriptions of Connected and Contrasting Images). Images are referenced by DOCCI filename (e.g., `train_00557.jpg`) and must be obtained from the original DOCCI dataset. ## Data Format ### SFT Configs (`sft_stage1`, `sft_stage2`, `sft_stage2_merged`) | Field | Type | Description | |-------|------|-------------| | `image` | `str` | DOCCI image filename (e.g., `train_00557.jpg`) | | `question` | `str` | Visual MCQ with `<image>` tag and answer choices | | `response` | `str` | Long CoT reasoning trace in `<think>...</think><answer>...</answer>` format | ### DPO Configs (`dpo_stage1`, `dpo_stage2`) | Field | Type | Description | |-------|------|-------------| | `image` | `str` | DOCCI image filename | | `question` | `str` | Visual MCQ with `<image>` tag and answer choices | | `chosen` | `str` | Preferred reasoning trace (correct answer) | | `rejected` | `str` | Dispreferred reasoning trace (incorrect answer) | ### Example (SFT) ```json { "image": "train_03916.jpg", "question": "<image>Which of the following correctly combines the number of cars in the left lane and the number of shorter, wider beams connecting the vertical red metal beam on the near side of the bridge?\nSelect from the following choices.\n(A) 1 car and 3 beams\n(B) 2 cars and 4 beams\n(C) 2 cars and 5 beams\n(D) 3 cars and 4 beams", "response": "<think> To solve this problem, let's break it down step by step:\n\n1. **Count the number of cars in the left lane:**\n - In the image, there are two cars visible in the left lane.\n\n2. **Count the number of shorter, wider beams connecting the vertical red metal beam on the near side of the bridge:**\n - Looking at the vertical red metal beam on the near side of the bridge, we can see that there are five shorter, wider beams connecting it to the horizontal structure above.\n\n3. **Combine the counts:**\n - We have 2 cars and 5 beams.\n\n... Wait, that doesn't seem right. Let me double-check the beams. The image shows four shorter, wider beams connecting the vertical poles on the near side. I must have miscounted earlier. Yes, it's four beams.\n\nSo the correct combination should be 2 cars and 4 beams. </think> <answer> ...." } ``` ## Usage ```python from datasets import load_dataset # Load a specific config sft_stage1 = load_dataset("nvidia/nemotron-research-LGT", "sft_stage1", split="train") sft_stage2 = load_dataset("nvidia/nemotron-research-LGT", "sft_stage2", split="train") # or stage2_merged # --- dpo_stage1 = load_dataset("nvidia/nemotron-research-LGT", "dpo_stage1", split="train") dpo_stage2 = load_dataset("nvidia/nemotron-research-LGT", "dpo_stage2", split="train") print(sft_stage1[0]) ``` ## Citation If you find this dataset or the LGT paper helpful, please cite: ```bibtex @article{acuna2025lgt, title={Long Grounded Thoughts: Synthesizing Visual Problems and Reasoning Chains at Scale}, author={Acuna, David and Yang, Chao-Han Huck and Deng, Yuntian and Jung, Jaehun and Lu, Ximing and Ammanabrolu, Prithviraj and Kim, Hyunwoo and Liao, Yuan-Hong and Choi, Yejin}, journal={arXiv preprint arXiv:2511.05705}, year={2026} } ```license: CC BY-NC 4.0 language: - en tags: - reasoning - vision - multimodal - vlm - chain-of-thought - nvidia size_categories: - 1M<n<10M configs: - config_name: sft_stage1 data_files: - split: train path: data/sft_stage1/train-*.parquet - config_name: sft_stage2 data_files: - split: train path: data/sft_stage2/train-*.parquet - config_name: sft_stage2_merged data_files: - split: train path: data/sft_stage2_merged/train-*.parquet - config_name: dpo_stage1 data_files: - split: train path: data/dpo_stage1/train-*.parquet - config_name: dpo_stage2 data_files: - split: train path: data/dpo_stage2/train-*.parquet <div align="center"> # LGT-1M+ **Long Grounded Thoughts: Synthesizing Visual Problems and Reasoning Chains at Scale** [](https://arxiv.org/abs/2511.05705) [](https://creativecommons.org/licenses/by-nc/4.0/) </div> **LGT（长锚定思维）**是一个合成视觉推理数据集，包含超过100万个复杂度各异的全新视觉推理问题，配套的推理轨迹展现了复杂的认知行为。该数据集覆盖视觉语言模型（Vision-Language Model, VLM）全流程的后训练场景：监督微调（Supervised Fine-Tuning, SFT）、离线强化学习（Direct Preference Optimization, DPO）以及在线强化学习（GRPO/RLVR）。 本数据集仅用于研发与学术研究。 ## Long Grounded Thoughts (LGT) 我们发现在当前以视觉为中心的推理数据合成框架向大规模扩展时，存在两大核心瓶颈：1) **问题合成饱和**——仅依赖字幕进行问题合成，会限制生成的视觉类问题的多样性与锚定性，导致下游性能快速饱和；2) **认知简化**——合成得到的推理轨迹仍缺乏高阶推理结构，部分原因在于底层问题过于简单，且合成过程多样性不足。 LGT针对这些核心挑战，重点聚焦**规模性**、**复杂度**与**思维链（Chain-of-Thought, CoT）丰富度**三大维度： - **规模性**：LGT通过利用锚定对象元数据，突破了仅依赖字幕的合成方法的瓶颈，将新型问题合成量从3万提升至超过100万高质量样本，且未出现性能饱和。 - **复杂度**：LGT大幅提升了视觉类问题合成的难度。其组合强化算法将易解答问题的占比降低了约10倍（从基线模型的36.7%降至3.3%）。 - **思维链丰富度**：LGT提升了合成推理轨迹的认知深度。该方法将复杂认知行为的出现频率提升了206%，最终得到的推理轨迹丰富度是基线模型的3倍。 ## 数据集概览 | 配置名称 | 拆分集 | 样本量 | 格式 | 描述 | |--------|-------|----------|--------|-------------| | `sft_stage1` | 训练集 | 753,675 | 监督微调（SFT） | 带有长思维链轨迹的锚定多项选择题（约75万条） | | `sft_stage2` | 训练集 | 163,873 | 监督微调（SFT） | 带有长思维链轨迹的组合强化型多项选择题 | | `sft_stage2_merged` | 训练集 | 243,873 | 监督微调（SFT） | 结合额外数据的第二阶段组合强化型多项选择题与长思维链轨迹 | | `dpo_stage1` | 训练集 | 129,318 | 直接偏好优化（DPO） | 来自第一阶段多项选择题的偏好样本对 | | `dpo_stage2` | 训练集 | 15,685 | 直接偏好优化（DPO） | 来自第二阶段组合强化型多项选择题的偏好样本对 | **源图像**：[DOCCI](https://google.github.io/docci/)（关联与对比图像描述数据集，Descriptions of Connected and Contrasting Images）。图像通过DOCCI文件名引用（例如`train_00557.jpg`），需从原始DOCCI数据集获取。 ## 数据格式 ### 监督微调配置（`sft_stage1`、`sft_stage2`、`sft_stage2_merged`） | 字段 | 类型 | 描述 | |-------|------|-------------| | `image` | 字符串 | DOCCI图像文件名（例如`train_00557.jpg`） | | `question` | 字符串 | 带有`<image>`标记与选项的视觉多项选择题 | | `response` | 字符串 | 采用`<think>...</think><answer>...</answer>`格式的长思维链推理轨迹 | ### 直接偏好优化配置（`dpo_stage1`、`dpo_stage2`） | 字段 | 类型 | 描述 | |-------|------|-------------| | `image` | 字符串 | DOCCI图像文件名 | | `question` | 字符串 | 带有`<image>`标记与选项的视觉多项选择题 | | `chosen` | 字符串 | 偏好的推理轨迹（正确答案版本） | | `rejected` | 字符串 | 非偏好的推理轨迹（错误答案版本） | ### 监督微调样例 json { "image": "train_03916.jpg", "question": "<image>Which of the following correctly combines the number of cars in the left lane and the number of shorter, wider beams connecting the vertical red metal beam on the near side of the bridge? Select from the following choices. (A) 1 car and 3 beams (B) 2 cars and 4 beams (C) 2 cars and 5 beams (D) 3 cars and 4 beams", "response": "<think> To solve this problem, let's break it down step by step: 1. **Count the number of cars in the left lane:** - In the image, there are two cars visible in the left lane. 2. **Count the number of shorter, wider beams connecting the vertical red metal beam on the near side of the bridge:** - Looking at the vertical red metal beam on the near side of the bridge, we can see that there are five shorter, wider beams connecting it to the horizontal structure above. 3. **Combine the counts:** - We have 2 cars and 5 beams. ... Wait, that doesn't seem right. Let me double-check the beams. The image shows four shorter, wider beams connecting the vertical poles on the near side. I must have miscounted earlier. Yes, it's four beams. So the correct combination should be 2 cars and 4 beams. </think> <answer> ...." } ## 使用方法 python from datasets import load_dataset # 加载指定配置 sft_stage1 = load_dataset("nvidia/nemotron-research-LGT", "sft_stage1", split="train") sft_stage2 = load_dataset("nvidia/nemotron-research-LGT", "sft_stage2", split="train") # 或加载stage2_merged配置 # --- dpo_stage1 = load_dataset("nvidia/nemotron-research-LGT", "dpo_stage1", split="train") dpo_stage2 = load_dataset("nvidia/nemotron-research-LGT", "dpo_stage2", split="train") print(sft_stage1[0]) ## 引用 如果您认为本数据集或LGT论文对您的研究有所帮助，请引用： bibtex @article{acuna2025lgt, title={Long Grounded Thoughts: Synthesizing Visual Problems and Reasoning Chains at Scale}, author={Acuna, David and Yang, Chao-Han Huck and Deng, Yuntian and Jung, Jaehun and Lu, Ximing and Ammanabrolu, Prithviraj and Kim, Hyunwoo and Liao, Yuan-Hong and Choi, Yejin}, journal={arXiv preprint arXiv:2511.05705}, year={2026} }