declare-lab/AlgoPuzzleVQA

AlgoPuzzleVQA数据集是为了挑战和评估多模态语言模型解决算法谜题的能力而设计的。这些谜题需要视觉理解、语言理解和复杂的算法推理。数据集中的谜题覆盖了多种数学和算法主题，如布尔逻辑、组合数学、图论、优化、搜索等，旨在评估视觉数据解释和算法问题解决技能之间的差距。数据集是通过自动化生成的，源自人类的代码，所有谜题都有确切的解决方案，无需繁琐的人工计算。这确保了数据集在推理复杂性和数据集大小方面可以无限扩展。研究显示，大型语言模型（LLM）如GPT4V和Gemini在谜题解决任务上的表现有限，它们的表现在多选题问答设置中接近随机水平。这些发现强调了在解决复杂推理问题时整合视觉、语言和算法知识的挑战。

The AlgoPuzzleVQA dataset is designed to challenge and evaluate the capabilities of multimodal language models in solving algorithmic puzzles that require both visual understanding, language understanding, and complex algorithmic reasoning. The puzzles cover a diverse array of mathematical and algorithmic topics such as boolean logic, combinatorics, graph theory, optimization, search, etc., aiming to assess the gap between visual data interpretation and algorithmic problem-solving skills. The dataset is automatically generated from code written by humans, ensuring that all puzzles have exact solutions that can be found without tedious human calculations, allowing for scalability in terms of reasoning complexity and dataset size. Research has shown that large language models (LLMs) such as GPT4V and Gemini exhibit limited performance in puzzle-solving tasks, with their performance approaching randomness in a multi-choice question-answering setup for a significant number of puzzles. The findings highlight the challenges of integrating visual, language, and algorithmic knowledge for solving complex reasoning problems.