STaD

# Dataset Card for STaD Scaffolded Benchmarks ## Dataset Summary The **STaD Scaffolded Benchmarks** are diagnostic evaluation datasets introduced in the paper *"STaD: Scaffolded Task Design for Identifying Compositional Skill Gaps in LLMs"* (ACL Findings 2026). These benchmarks go beyond measuring whether a model answers correctly — they reveal **where and why** a model fails by breaking problems into sub-tasks and providing targeted scaffolding at specific reasoning steps. Each sample includes: - The original benchmark problem and answer - A decomposition into **sub-tasks**, each mapped to a specific reasoning skill - **Scaffolded variations** of the original problem — versions where partial solutions are provided up to a certain step — enabling pinpoint diagnosis of reasoning breakdowns - **Decomposed sub-questions** that isolate individual reasoning steps The benchmarks cover three source datasets: GSM8K, MATH-Hard, and Tree-of-Thought (ToT) Arithmetic. --- ## Datasets | Source Benchmark | Description | |-----------------|-------------| | GSM8K | Grade school math word problems with scaffolded variations | | MATH (Hard) | Competition-level math problems across 6 categories | | ToT Arithmetic | Multi-step arithmetic problems from Tree-of-Thought | ## Configurations | Config | # Samples | Description | |--------|-----------|-------------| | `default` | 1,005 | A curated subset containing only the highest-quality decompositions and scaffolding. Recommended for most diagnostic work and loaded by default. | | `acl_2026` | 3,397 | The full set of samples as used in the ACL Findings 2026 paper. | --- ## Data Fields | Field | Type | Description | |-------|------|-------------| | `id` | string | Unique identifier for the sample | | `question` | string | Original problem statement | | `answer` | string | Ground truth final answer | | `solution` | string | Full reference solution (MATH-Hard only) | | `category` | string | Problem category (MATH-Hard only; e.g., Algebra, Number Theory) | | `sub-task` | list | List of reasoning steps, each with a `segment` (description) and `skill` (reasoning skill label) | | `sub-task-answer` | list | Step-by-step answers with `explanation` and `answer` for each sub-task | | `scaffolding` | list | Scaffolded problem variants — each provides partial solutions up to step *k*, leaving the rest for the model | | `scaffolding_verification` | list | Verification data for scaffolded variants | | `decompositions` | list | Sub-questions that decompose the original problem into individual reasoning steps | --- ## Intended Uses - **Diagnostic evaluation**: Identify *where* in the reasoning chain a model breaks down, not just whether it fails - **Skill gap analysis**: Pinpoint which reasoning skills (e.g., modular arithmetic, multi-step algebra) a model lacks - **Compositional reasoning research**: Study how models handle problems requiring multiple skills in combination - **Benchmarking**: Compare LLMs at a fine-grained, sub-task level rather than aggregate accuracy --- ## How to Use To run evaluations using this dataset, refer to the code and instructions in the GitHub repository: **[https://github.com/ibm-granite/scaffolded-task-design](https://github.com/ibm-granite/scaffolded-task-design)** ```python from datasets import load_dataset # Recommended: high-confidence subset (loads by default) ds = load_dataset("ibm-research/STaD") # Exact replication of ACL Findings 2026 ds_full = load_dataset("ibm-research/STaD", "acl_2026") # Filter to a single source benchmark via the `dataset` field gsm8k = ds["test"].filter(lambda x: x["dataset"] == "gsm8k") ``` ## Limitations - **Evaluation only**: With ~3K samples, these datasets are designed for evaluation and diagnostic purposes, not large-scale training. - **Math domain**: The current benchmarks focus on mathematical reasoning. Extension to other domains is left for future work. - **Scaffolding assumes step ordering**: The scaffolded variations assume a fixed sub-task order; alternative decompositions may yield different results. --- ## How to Cite If you use this dataset in your research, please cite: ```bibtex @inproceedings{an2026stad, title={STaD: Scaffolded Task Design for Identifying Compositional Skill Gaps in LLMs}, author={An, Sungeun and Kadhe, Swanand Ravindra and Thakur, Shailja and DeLuca, Chad and Patel, Hima}, booktitle={Findings of the Association for Computational Linguistics: ACL 2026}, pages={39675--39705}, year={2026} } ```