SteelQA v1.0 — A Machine-Scorable QA & Refusal Benchmark for Steel-Metallurgy LLMs

SteelQA is a machine-scorable question-answering and refusal benchmark for large language models adapted to the steel-metallurgy domain. It contains 1,118 questions across 10 categories (composition recall, property reasoning, cross-standard equivalence, heat treatment, specification compliance, hallucination traps, Russian-language items, multi-step reasoning, and open-ended exploration). Every question ships with an explicit, machine-readable rubric (expected_topics, must_not_say, ground_truth_numbers), so evaluation requires no LLM judge: the included reference scorer (eval_scoring.py, pure-Python) turns each model answer into a score in [0, 1]. A numeric answer is matched as a point value (within ±15%) or as a [min, max] range. Numeric ground-truth is never fabricated: it is derived verbatim from SteelBench v1.0 (1,636 heats / 594 grades). 883 questions are templated over real SteelBench records (named-grade composition/strength recall and composition-to-property prediction); 235 are expert-curated. Only unambiguous-unit fields are admitted as ground-truth (tensile/yield strength in MPa, composition in wt%, temperatures in degrees C); hardness is excluded because the source mixes unlabeled HRC/HRB/HV/HB scales. The release is self-contained and reproducible: data in JSON/JSONL/CSV, the reference scorer, a datasheet (DATASET_CARD.md), a filled ML Reproducibility Checklist v2.0 (REPRODUCIBILITY.md), Croissant metadata, SHA-256 checksums, and a code/ directory that regenerates the benchmark from source. A build-time invariant guarantees that a perfect synthetic answer scores 1.0 on every record. Companion to SteelBench v1.0. Intended as an evaluation-only benchmark (no training split). Known limitations: small Russian coverage (30 of 1,118) and single-domain scope (steel); single-seed reference results in the accompanying paper.