TIGER-Lab/SWE-QA-Pro-Bench

SWE-QA-Pro Bench 是一个存储库级别的问答数据集，旨在评估模型是否能在真实世界代码库上进行基于实际、自主推理的问答。与先前专注于流行存储库或短代码片段的基准不同，SWE-QA-Pro 强调：具有多样化结构和领域的长尾存储库；需要导航多个文件的存储库基础问题；以及自主推理，其中模型必须探索代码而非依赖记忆知识。该数据集通过数据驱动流程构建：收集大规模GitHub问题并将其组织成涵盖多样化软件工程任务的48个语义集群；合成基于可执行存储库的问答对；应用难度校准步骤以移除无需存储库交互即可解决的问题。最终数据集包含来自26个存储库的260个高质量问答对（每个存储库10个），解决任务通常需要多步推理和代码库探索。

SWE-QA-Pro Bench is a repository-level question answering dataset designed to evaluate whether models can perform grounded, agentic reasoning over real-world codebases. Unlike prior benchmarks that focus on popular repositories or short code snippets, SWE-QA-Pro emphasizes: long-tail repositories with diverse structures and domains; repository-grounded questions that require navigating multiple files; and agentic reasoning, where models must explore code rather than rely on memorized knowledge. The dataset is constructed through a data-driven pipeline: collecting large-scale GitHub issues and organizing them into 48 semantic clusters covering diverse software engineering tasks; synthesizing QA pairs grounded in executable repositories; and applying a difficulty calibration step to remove questions solvable without repository interaction. The final dataset contains 260 high-quality QA pairs from 26 repositories (10 per repository), where solving tasks typically requires multi-step reasoning and codebase exploration.