STEMBench

Task data for a benchmark of LLM agents on research-grade STEM tasks across astrophysics, high-energy physics, and neuroscience. The harness that consumes this data is deposited as a companion software record (see Related identifiers). CONTENTS One archive per task payload, plus a documentation-only bundle of task descriptions (statements, scoring definitions, dataset notes, DOI-based literature indexes). Each archive unpacks at the repository root, so a payload lands in tasks/<domain>/<task_id>/dataset/ — a public split staged into the agent's workspace, and a private split with the held-out ground truth used by the grader. Every file is covered by per-task SHA-256 checksums shipped in the software record. Verify a payload with: python scripts/verify_data.py <domain>/<task_id> 11 of the 12 task payloads are published here. NOT INCLUDED ABIDE-derived data. The source is CC-BY-NC-SA and derives from human neuroimaging data, so nothing ABIDE-derived is redistributed. The software record ships a script that rebuilds the payload locally and verifies it a published checksums. The HiggsML test input, public/pseudo_experiments.parquet (5.56 GB). It is regenerated from the upstream CC-BY release (doi:10.5281/zenodo.15131565) with the generator and exact build parameters recorded in the payload's manifest.json. Instructions travel inside the archive as higgs_note.txt. LICENCE AND ATTRIBUTION Released under CC-BY-4.0. Synthetic payloads were generated for the Payloads derived from external sources carry their original attribution, which is a licence condition: LHC Olympics 2020, doi:10.5281/zenodo.4287846Top Quark Tagging Reference Dataset (Kasieczka, Plehn, Thompson, Russel), doi:10.5281/zenodo.2603256DarkMachines secret dataset, doi:10.5281/zenodo.4443151PLAsTiCC unblinded data (PLAsTiCC Team and Modelers), doi:10.5281/zenodo.2539456FAIR Universe HiggsML Uncertainty Challenge Public Dataset, doi:10.5281/zenodo.15131565 One task derives from Hinode/SOT-SP observations obtained through Hinode/DARTS archive (ISAS/JAXA and NAOJ), with Milne-Eddington targets from MERLIN inversions by the Community Spectro-polarimetric Analysis Center (CSAC) at HAO/NCAR. The archive asks for acknowledgment rather than imposing licence terms: "Hinode is a Japanese mission developed and launched by ISAS/JAXA, with NAOJ as domestic partner and NASA and STFC (UK) as international partners. It is operated by these agencies in co-operation with ESA and NSC (Norway)."