OpenMOSS-Team/VideoThinkBench
收藏资源简介：
--- language: - en task_categories: - text-to-video - image-to-video - visual-question-answering - question-answering license: mit dataset_info: - config_name: Eyeballing_Puzzles features: - name: task dtype: string - name: id dtype: string - name: prompt dtype: string - name: image dtype: image - name: solution_image dtype: image - config_name: Visual_Puzzles features: - name: task dtype: string - name: id dtype: string - name: prompt dtype: string - name: image dtype: image - name: solution_image dtype: image - config_name: ARC_AGI_2 features: - name: task dtype: string - name: id dtype: string - name: prompt dtype: string - name: image dtype: image - name: solution_image dtype: image - config_name: Mazes features: - name: task dtype: string - name: id dtype: string - name: prompt dtype: string - name: image dtype: image - name: solution_image dtype: image - config_name: Text_Centric_Tasks features: - name: task dtype: string - name: id dtype: string - name: prompt dtype: string - name: image dtype: image - name: answer dtype: string configs: - config_name: Eyeballing_Puzzles data_files: - split: minitest path: minitest_Vision-Centric_Reasoning/Eyeballing_Puzzles.parquet - split: test path: Vision-Centric_Reasoning/Eyeballing_Puzzles.parquet - config_name: Visual_Puzzles data_files: - split: minitest path: minitest_Vision-Centric_Reasoning/Visual_Puzzles.parquet - split: test path: Vision-Centric_Reasoning/Visual_Puzzles.parquet - config_name: ARC_AGI_2 data_files: - split: minitest path: minitest_Vision-Centric_Reasoning/ARC-AGI-2.parquet - split: test path: Vision-Centric_Reasoning/ARC-AGI-2.parquet - config_name: Mazes data_files: - split: minitest path: minitest_Vision-Centric_Reasoning/Mazes.parquet - split: test path: Vision-Centric_Reasoning/Mazes.parquet - config_name: Text_Centric_Tasks data_files: - split: minitest path: minitest_Text-Centric_Reasoning/Text-Centric.parquet - split: test path: Text-Centric_Reasoning/Text-Centric.parquet --- <div align="center"> # [CVPR 2026] Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm </div> <div align="center" style="font-size: 15pt"> <a href='https://arxiv.org/abs/2511.04570'><img src='https://img.shields.io/badge/Arxiv-2511.04570-purple'></a> <a href='https://huggingface.co/papers/2511.04570'><img src='https://img.shields.io/badge/HF%20Paper-2511.04570-blue'></a> <a href='https://thinking-with-video.github.io/'><img src='https://img.shields.io/badge/Project-Website-green'></a> <a href='https://github.com/tongjingqi/Thinking-with-Video'><img src='https://img.shields.io/badge/Code-GitHub-black'></a> <a href='https://thinking-with-video.github.io/#leaderboard'><img src='https://img.shields.io/badge/Leaderboard-Table-E07A5F'></a> </div> <div align="center"> <a href="https://huggingface.co/papers/month/2025-11"> <img src="assets/huggingface_paper_gold_month.svg"/> </a> </div> ## 🎊 News <!-- omit in toc --> - [2026.02] 🔥🔥*Our work has been accepted by* **CVPR 2026**! 🎉🎉🎉 - [2025.11] Our paper "Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm" has been released on arXiv! 📄 [[Paper](https://arxiv.org/abs/2511.04570)] On HuggingFace, it has achieved "#1 Paper of the Day"! - [2025.11] 🔥We release *["minitest"](https://huggingface.co/datasets/OpenMOSS-Team/VideoThinkBench)* of our VideoThinkBench, including 500 test samples of vision-centric tasks and 250 test samples of text-centric tasks. - [2025.12] 🔥We release VideoThinkBench [Leaderboard](https://thinking-with-video.github.io/#leaderboard) that includes different models. ## 📜 Brief Introduction <!-- omit in toc --> Moving beyond the traditional paradigms of "Thinking with Text" (e.g., Chain-of-Thought) and "Thinking with Images", we propose "**Thinking with Video**"—a new paradigm that unifies visual and textual reasoning through video generation models. It naturally enables human-like dynamic reasoning through video generation, such as **drawing and imagination**. 💡 **A New Unified Reasoning Paradigm** "Thinking with Video" leverages video generation models to visualize dynamic processes, represent temporal evolution, and embed text within video frames. This approach achieves unified multimodal understanding and generation, overcoming the static constraints of image-based reasoning and the modality separation in traditional approaches. 📊 **VideoThinkBench: A Comprehensive Benchmark** We developed VideoThinkBench, the first reasoning benchmark specifically designed for evaluating video generation models. It comprises vision-centric tasks (eyeballing puzzles, visual puzzles, ARC-AGI-2, mazes) that leverage dynamic visual reasoning, and text-centric tasks adapted from established benchmarks (MATH, GSM8K, MMLU, MMMU, etc.) that test text-based reasoning capabilities within generated videos. 🚀 **Surpassing VLMs on Several Tasks** Our evaluation shows that Sora-2 demonstrates competitive reasoning capabilities across both categories. Notably, Sora-2 **surpasses state-of-the-art vision-language models on several vision-centric tasks**, showcasing the unique advantages of dynamic visual reasoning. On text-centric tasks, Sora-2 achieves strong performance including 98.9% on GSM8K, 94.0% on MATH, and 75.5% on MMMU, demonstrating the potential of "Thinking with Video" as a unified multimodal reasoning paradigm. <div align="center"> <img src="assets/main_picture.png" width=80% /> </div> ## 📝 Paper Abstract <!-- omit in toc --> "Thinking with Text" and "Thinking with Images" paradigm significantly improve the reasoning ability of large language models (LLMs) and Vision Language Models (VLMs). However, these paradigms have inherent limitations. (1) Images capture only single moments and fail to represent dynamic processes or continuous changes, and (2) The separation of text and vision as distinct modalities, hindering unified multimodal understanding and generation. To overcome these limitations, we introduce "Thinking with Video", a new paradigm that leverages video generation models, such as Sora-2, to bridge visual and textual reasoning in a unified temporal framework. To support this exploration, we developed the Video Thinking Benchmark (VideoThinkBench). VideoThinkBench encompasses two task categories: (1) vision-centric tasks (e.g., Eyeballing Puzzles), and (2) text-centric tasks (e.g., subsets of GSM8K, MMMU). Our evaluation establishes Sora-2 as a capable reasoner. On vision-centric tasks, Sora-2 is generally comparable to state-of-the-art (SOTA) VLMs, and even surpasses VLMs on several tasks, such as Eyeballing Games. On text-centric tasks, Sora-2 achieves 92% accuracy on MATH, and 75.53% accuracy on MMMU. Furthermore, we systematically analyse the source of these abilities. We also find that self-consistency and in-context learning can improve Sora-2's performance. In summary, our findings demonstrate that the video generation model is the potential unified multimodal understanding and generation model, positions "thinking with video" as a unified multimodal reasoning paradigm. ## 📚 VideoThinkBench Details <!-- omit in toc --> VideoThinkBench is a comprehensive benchmark for evaluating video generation models' reasoning capabilities, consisting of two main categories: ### Vision-Centric Tasks - **Eyeballing Puzzles**: Spatial reasoning tasks requiring visual estimation and drawing - **Visual Puzzles**: Pattern recognition and visual logic problems - **ARC-AGI-2**: Abstract reasoning tasks requiring few-shot learning - **Mazes**: Path-finding and navigation challenges ### Text-Centric Tasks Adapted from established benchmarks including: - **Mathematical Reasoning**: MATH, GSM8K, AIME, MathVista, MathVision - **Multimodal Understanding**: MMMU, MMBench - **General Knowledge**: MMLU, MMLU-Pro - **Scientific Reasoning**: GPQA-diamond, SuperGPQA <!-- Dataset is available on [Hugging Face](https://huggingface.co/datasets/fnlp/VideoThinkBench). --> ## ✨ Benchmark Results <!-- omit in toc --> ### Performance Comparison Across All Tasks The table below summarizes the accuracy (%) of Sora-2 compared with state-of-the-art vision-language models across all second-level tasks in VideoThinkBench: | **Category** | **Task** | **Sora-2** | **Gemini 2.5 Pro** | **GPT5 high** | **Claude Sonnet 4.5** | |--------------|----------|------------|-------------------|--------------|---------------------| | **Vision-Centric** | Eyeballing-Point | 44.7 | 27.8 | 33.6 | 36.2 | | | Eyeballing-Line | 38.0 | 21.0 | 24.0 | 26.3 | | | Eyeballing-Shape | 34.5 | 34.5 | 32.5 | 50.5 | | | Visual-Color | 67.0 | 73.9 | 79.6 | 85.6 | | | Visual-Shape | 64.9 | 92.9 | 97.5 | 68.6 | | | ARC-AGI-2 | 1.3 | 4.9 | 9.9 | 13.6 | | | **Average** | **41.7** | **42.5** | **46.2** | **46.8** | | **Text-Centric** | Text-Only Math | 53.6 | 94.8 | 97.2 | 90.0 | | | Text-Only General Knowledge | 63.1 | 84.5 | 85.2 | 86.3 | | | Multimodal Math | 56.3 | 66.7 | 69.6 | 65.6 | | | Multimodal General Knowledge | 49.4 | 83.0 | 80.6 | 82.3 | | | **Average** | **55.6** | **82.3** | **83.2** | **81.1** | | **Overall Average** | | **47.3** | **58.4** | **61.0** | **60.5** | **Note**: For Sora-2: Eyeballing Puzzles use Major Frame evaluation; Visual Puzzles show the average of Color-Filling and Shape-Drawing tasks; Text-Centric Reasoning tasks use Video evaluation results. **🔥Leaderboard: [HERE](https://thinking-with-video.github.io/#leaderboard)** <!-- ## 💻 Sample Usage (Evaluation) --> <!-- To evaluate models on the tasks within VideoThinkBench, you can use the provided scripts from the accompanying GitHub repository. --> <!-- First, you need to clone the repository and install the dependencies: --> <!-- ```bash --> <!-- git clone https://github.com/tongjingqi/Thinking-with-Video.git --> <!-- cd Thinking-with-Video --> <!-- pip install -r requirements.txt --> <!-- ``` --> <!-- *(Note: Installation dependencies are listed as "Coming soon" in the GitHub README, but the command is provided.)* --> <!-- Then, you can run the evaluation scripts for different task categories: --> <!-- ```bash --> <!-- # Vision-centric tasks evaluation --> <!-- python eval_vision_centric.py --task eyeballing --> <!-- # Text-centric tasks evaluation --> <!-- python eval_text_centric.py --benchmark MATH --> <!-- ``` --> ## 💡 Takeaways <!-- omit in toc --> Our systematic evaluation on VideoThinkBench reveals seven key findings: 1. **Surpassing VLMs on Eyeballing Puzzles**: Sora-2 generally **surpasses SOTA VLMs** on eyeballing puzzles, exhibiting strong **geometric and physical reasoning** abilities. It can simulate the extension and reflection of rays and manipulate geometric elements (e.g., points and lines) to support spatial reasoning. 2. **Inductive Reasoning on Visual Puzzles**: Sora-2's performance is comparable to Claude Sonnet 4.5 on Shape-Drawing puzzles, demonstrating **inductive reasoning** capabilities. Sora-2 can recognize and apply **patterns of color, shape, and size**, solving visual puzzles involving symmetry, gradients, and compositionality. 3. **Few-Shot Learning Capabilities**: **Sora-2 is a few-shot learner**. On ARC-AGI-2, which requires finding patterns in input-output pairs, while SOTA VLMs achieve less than 5% accuracy, Sora-2 can often make **reasonable predictions**, although they do not strictly match dataset annotations. 4. **Unified Multimodal Reasoning**: On text-centric tasks, Sora-2 shows surprising performance on text and multimodal reasoning benchmarks. The video generation model can **embed text within video frames**, enabling unified multimodal understanding and generation. This demonstrates that "Thinking with Video" is potentially a **unified multimodal reasoning paradigm**. 5. **Improved In-Context Learning with More Examples**: Sora-2 achieves better in-context learning by providing more examples. Experiments show that Sora-2 performs better when provided with all examples compared to only one example, revealing an underexplored direction for analyzing and improving the in-context learning abilities of video generation models. 6. **Test-Time Scaling with Self-Consistency**: **Self-consistency can improve** Sora-2's performance on verifiable video generation reasoning tasks. This reveals an underexplored direction: **test-time scaling in video generation reasoning tasks**. 7. **Analysis of Capability Source**: We systematically analyzed the **source of Sora-2's capabilities**. Sora-2 maintains performance comparable to the original test set on adapted math problems, reducing the likelihood of test set leakage. However, Sora-2 struggles to generate coherent reasoning processes in videos, even when providing correct final answers. Through comparative experiments with Wan 2.5, we speculate that Sora-2's text-centric reasoning ability originates from its **prompt rewriter** model. ## ⚖️ Licenses <!-- omit in toc --> [](LICENSE) This project is licensed under the MIT License - see the LICENSE file for details. ## 🔎 Citation If you find our work helpful, please consider citing our paper 📝 and starring us ⭐️! ```bibtex @inproceedings{tong2026thinking, title={Thinking with video: Video generation as a promising multimodal reasoning paradigm}, author={Tong, Jingqi and Mou, Yurong and Li, Hangcheng and Li, Mingzhe and Yang, Yongzhuo and Zhang, Ming and Chen, Qiguang and Liang, Tianyi and Hu, Xiaomeng and Zheng, Yining and others}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages={41121--41129}, year={2026} } ``` --- <div align="center"> Made with ❤️ for advancing multimodal reasoning research </div>
VideoThinkBench is a comprehensive benchmark dataset designed to evaluate the reasoning capabilities of video generation models. It consists of two main categories of tasks: vision-centric tasks and text-centric tasks. Vision-centric tasks include Eyeballing Puzzles (spatial reasoning tasks), Visual Puzzles (pattern recognition and visual logic problems), ARC-AGI-2 (abstract reasoning tasks), and Mazes (path-finding and navigation challenges). Text-centric tasks are adapted from established benchmarks such as MATH, GSM8K, MMLU, and MMMU, covering areas like mathematical reasoning, multimodal understanding, general knowledge, and scientific reasoning. The dataset aims to unify visual and textual reasoning through video generation models, overcoming the limitations of traditional image and text-based reasoning paradigms.
