nostudynoworking/R1-Onevision

--- viewer: true license: apache-2.0 task_categories: - question-answering size_categories: - 100K<n<1M --- ## R1-Onevision [\[📂 GitHub\]](https://github.com/Fancy-MLLM/R1-Onevision)[\[📝 Paper\]](https://arxiv.org/pdf/2503.10615) [\[🤗 Reasoning Benchmark\]](https://huggingface.co/datasets/Fancy-MLLM/R1-OneVision-Bench) [\[🤗 HF Demo\]](https://huggingface.co/spaces/Fancy-MLLM/R1-OneVision) # R1-Onevision Dataset ## Dataset Overview The **R1-Onevision** dataset is a meticulously crafted resource designed to empower models with advanced multimodal reasoning capabilities. Aimed at bridging the gap between visual and textual understanding, this dataset provides rich, context-aware reasoning tasks across diverse domains, including natural scenes, science, mathematical problems, OCR-based content, and complex charts. It combines high-quality data from LLaVA-OneVision with domain-specific datasets, each carefully selected and filtered to provide a solid foundation for complex visual reasoning tasks. With a focus on enabling deep reasoning and accurate model predictions, **R1-Onevision** equips models to handle a variety of visual and textual inputs, tackling intricate reasoning challenges with precision. # Data Generation Pipeline ## The Genesis: Data Preparation & Filtering R1-Onevision began with the aggregation of a diverse collection of datasets spanning natural images, OCR, charts, science, mathematical problems, and more. These datasets were filtered and curated with the core objective of supporting reasoning tasks across different domains. We ensured that only the most relevant data and question types were selected, setting the stage for a dataset that offers both quality and depth. In addition to LLaVA-OneVision, the dataset incorporates a carefully chosen set of domain-specific datasets, enhancing the robustness of the dataset and making it an ideal tool for multimodal reasoning research. ## Image Captioning: Turning Pixels into Formal Language Captions serve as the backbone of this dataset, capturing fine-grained visual details to enable deep reasoning. The process involved the combined power of GPT-4o, Grounding DINO, and EasyOCR, which allowed us to generate detailed captions and spatial information for a variety of images. - **Charts & Diagrams**: GPT-4o was employed to translate visual elements into structured formats, including SPICE for circuit schematics, PlantUML for flowcharts, and HTML for UI layouts. These captions also provided explanations in a formal language, aiding in their use for reasoning tasks. - **Natural Scenes**: GPT-4o generated rich, descriptive captions for images, while Grounding DINO pinpointed the key elements within the scene to enhance model understanding. - **Text-Only Images** : EasyOCR extracted text from images containing printed or handwritten text, and GPT-4o restored the original format to ensure that context and layout were preserved. - **Images with Textual Content**: For images containing both visual and textual elements, OCR data, bounding boxes, and GPT-4o-generated captions combined to recreate the original layout and structure. - **Mathematical Images**: For images involving mathematical content, GPT-4o summarized the image content into structured captions, reasoning steps, and results, ensuring that the context provided could directly support complex reasoning tasks. ## Reasoning Process: Chain-of-Thought Generation Once captions were generated, the next step involved reasoning over the images. The **Chain-of-Thought (CoT)** approach was used to guide the model through a structured reasoning process, drawing on both textual and visual information. To enhance the model’s reasoning ability, we implemented a **Role-Playing** approach. This allowed the model to "see" the image and iteratively revisit key visual information to refine its understanding and reasoning. This process encouraged the model to think more critically about the visual elements, generating more accurate and contextually rich answers. ## Final Filter: Quality Assurance The dataset also includes a final layer of quality assurance. GPT-4 was used to filter out any inaccurate or irrelevant reasoning steps, ensuring that only valid, coherent, and contextually accurate conclusions remained. This layer of validation strengthens the reliability and trustworthiness of the reasoning process. # Data Format The data is stored in `parquet` files with the following structure: ```json { "id": "<unique identifier>", "image": "<base64>", "conversations": [ {"from": "human", "value": "<question>"}, {"from": "assistant", "value": "<cot>"} ] } ``` # Data distribution <img src="https://cdn-uploads.huggingface.co/production/uploads/65af78bb3e82498d4c65ed2a/W4IG0lu2BrXqwXRIXwdzL.png"/> # Institution - Zhejiang University # Dataset Contact - [e-mail] - [e-mail] - [e-mail]数据集元信息： - 支持预览（viewer: true） - 许可证：Apache-2.0 - 任务类别：问答（question-answering） - 数据规模：10万~100万条（100K<n<1M） --- ## R1-Onevision [📂 GitHub仓库](https://github.com/Fancy-MLLM/R1-Onevision) [📝 论文](https://arxiv.org/pdf/2503.10615) [🤗 推理基准集](https://huggingface.co/datasets/Fancy-MLLM/R1-OneVision-Bench) [🤗 Hugging Face 演示](https://huggingface.co/spaces/Fancy-MLLM/R1-OneVision) # R1-Onevision 数据集 ## 数据集概览 **R1-Onevision 数据集**是一款精心打造的多模态推理能力增强型资源，旨在弥合视觉与文本理解之间的鸿沟。该数据集涵盖自然场景、科学、数学问题、基于光学字符识别（Optical Character Recognition, OCR）的内容以及复杂图表等多元领域，提供丰富的上下文感知推理任务。 它整合了LLaVA-OneVision的高质量数据与领域专用数据集，所有数据均经过严格筛选与预处理，为复杂视觉推理任务奠定坚实基础。本数据集聚焦于支持深度推理与精准模型预测，能够赋能模型处理多样的视觉与文本输入，精准应对复杂的推理挑战。 # 数据生成流程 ## 起源：数据准备与筛选 R1-Onevision 首先整合了涵盖自然图像、OCR、图表、科学、数学问题等多个领域的多样化数据集。所有数据集均经过筛选与整理，核心目标是支持跨领域的推理任务。我们仅选择最相关的数据与问题类型，确保数据集兼具质量与深度。 除LLaVA-OneVision外，本数据集还纳入了精心挑选的领域专用数据集，提升了数据集的鲁棒性，使其成为多模态推理研究的理想工具。 ## 图像字幕生成：将像素转化为规范语言 字幕是本数据集的核心支撑，能够捕捉细粒度的视觉细节以支持深度推理。该流程结合了GPT-4o、Grounding DINO与EasyOCR的能力，可为各类图像生成详细的字幕与空间信息。 - **图表与示意图**：使用GPT-4o将视觉元素转化为结构化格式，包括用于电路原理图的SPICE、用于流程图的PlantUML以及用于UI布局的HTML。此类字幕同时辅以规范语言的解释，便于在推理任务中使用。 - **自然场景**：GPT-4o为图像生成丰富的描述性字幕，同时Grounding DINO可精准定位场景中的关键元素，增强模型的理解能力。 - **纯文本图像**：EasyOCR从包含印刷或手写文本的图像中提取文本，随后GPT-4o还原原始格式，确保上下文与布局得以保留。 - **含文本内容的图像**：对于同时包含视觉与文本元素的图像，将OCR数据、边界框与GPT-4o生成的字幕相结合，还原原始布局与结构。 - **数学图像**：针对包含数学内容的图像，GPT-4o将图像内容总结为结构化字幕、推理步骤与结果，确保提供的上下文可直接支持复杂推理任务。 ## 推理过程：思维链（Chain-of-Thought, CoT）生成 完成字幕生成后，下一步是对图像进行推理。我们采用**思维链（Chain-of-Thought, CoT）**方法，引导模型通过结构化推理过程，同时利用文本与视觉信息。 为进一步提升模型的推理能力，我们引入了**角色扮演（Role-Playing）**方法。该方法允许模型“查看”图像并反复回顾关键视觉信息，以优化其理解与推理过程，鼓励模型对视觉元素进行更严谨的思考，生成更精准且上下文丰富的答案。 ## 最终筛选：质量保证 本数据集还包含一层最终的质量验证流程。使用GPT-4过滤掉所有不准确或无关的推理步骤，仅保留有效、连贯且上下文准确的结论。此验证环节强化了推理过程的可靠性与可信度。 # 数据格式 数据以`parquet`文件存储，结构如下： json { "id": "<唯一标识符>", "image": "<base64编码的图像数据>", "conversations": [ {"from": "human", "value": "<问题>"}, {"from": "assistant", "value": "<思维链内容>"} ] } # 数据分布  # 所属机构 - 浙江大学 # 数据集联系方式 - [e-mail] - [e-mail] - [e-mail]