BenGER

BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law BenGER is a benchmark for evaluating Large Language Models on the kind of long-form, doctrinal-rubric-graded legal reasoning that German law distribution of the dataset. The dataset — one benchmark, three subsets Benchathon — 15 exam-style tasks at intermediate difficulty, each with multiple human-written solutions (unaided and human–AI co-creation conditions). A controlled validation subset received 180 blind human reviews from seven domain experts; their grading is shipped alongside the LLM-judge scoring. ZJS — 581 published exam-style cases from the Zeitschrift für das Juristische Studium (2008–2026), spanning civil, criminal, and public law. Case texts and reference solutions are released for the 198 cases whose authors granted IP clearance; for the remaining 383 the Aufgabe and Musterlösung fields point to the ZJS search interface (https://www.zjs-online.com/index.php?sektion=3) so readers can fetch the originals. Grundprinzipien — 531 short doctrinal-reasoning items (Ja/Nein decisions with explanatory reasoning) covering foundational principles across the three legal branches. In total: 12 contemporary LLM systems (closed-flagship, efficiency-tier, and open-weight) evaluated on these three subsets, with a rubric-aligned LLM-as-a-Judge cross-validated against a multi-rater human-grading protocol. Anonymisation Benchathon participants are referenced by pseudonyms (AblePartner, ActiveExplorer, …). The seven human graders are referred to by stable codes grader_01..grader_07 across every released file. Companion artefacts Source code, manuscript, analysis pipeline, anonymisation scripts: https://github.com/SebastianNagl/benger-platform arXiv preprint: t.b.d. Companion ICAIL system-demonstration paper - the BenGER platform that produced this dataset: Nagl & Grabmair, ICAIL 2026 — see the GitHub repository for full citation details. License CC BY 4.0. When you use this dataset, please cite the companion paper (arXiv preprint forthcoming) and link back to this Zenodo record.