Magistrate-Bench

Note: This is a partial release of the Magistrate Bench dataset. The complete dataset will be uploaded in a subsequent version. Magistrate-Bench: A Multilingual, Repository-Context Aware Code Review Dataset Magistrate-Bench is a large-scale benchmark dataset designed to evaluate automated code review agents in realistic, repository-level scenarios. Unlike existing datasets that focus on isolated code snippets or single-file diffs, Magistrate-Bench provides the full repository context, enabling the evaluation of complex, cross-file reasoning capabilities. 📊 Dataset Statistics Total Pull Requests: 2,042 Total Issues: 12,121 (Avg. ~6 per PR) Source Repositories: 245 active open-source projects (>500 stars) Languages: 12 🌍 Language Distribution The dataset covers a diverse range of programming languages to ensure broad generalization: Python: 389 PRs JavaScript: 272 PRs TypeScript: 232 PRs Rust: 207 PRs Java: 186 PRs Go: 182 PRs PHP: 105 PRs Dart: 99 PRs C#: 98 PRs Swift: 96 PRs C++: 90 PRs Ruby: 86 PRs 🔑 Key Features Repository-Context Aware: Each PR is paired with a cached snapshot of the entire repository state at the moment of review (specifically, the commit immediately preceding the first human review). This allows agents to trace dependencies, imports, and definitions across files. Real-World Ground Truth: Derived from merged PRs with rigorous human review threads (min. 6 threads per PR). Temporal Consistency: We algorithmically reconstruct the exact state of the codebase that the human reviewer saw, ensuring that the "ground truth" comments align perfectly with the code version being analyzed. Bot & Noise Filtering: All automated bot comments (e.g., CodeCov, Dependabot) and "LGTM" noise are strictly filtered out to focus on substantive engineering feedback. Standardized Taxonomy: Issues are classified into standardized categories: Style: 4,969 issues Design: 2,523 issues Documentation: 1,931 issues Logic: 1,120 issues Bug: 973 issues 🛠 Construction Methodology The dataset was built using a hybrid extraction pipeline: Selection: Targeted active repositories with high contributor activity. Extraction: Scraped PR metadata, diffs, and review threads via GitHub API. Classification: Used an LLM-based classifier (T=0) to categorize free-text review comments into structured issue types. Verification: Programmatically cross-referenced every extracted issue against the actual diff to strictly validate file paths and line numbers.