Terminal-Lego-15k

> [!NOTE] > This private PrimeIntellect mirror is filtered from > [`SWE-Lego/Terminal-Lego-15k`](https://huggingface.co/datasets/SWE-Lego/Terminal-Lego-15k). > It reflects `ad6e280` from [prime-data PR #34](https://github.com/PrimeIntellect-ai/prime-data/pull/34): 1,224 > tasks are excluded and 13,825 task directories remain. > > Each retained task's `task.toml` sets `[environment].docker_image` to its > prebuilt Prime platform image in the canonical org-less form > `terminal-lego/<task-id>:latest` (2026-07 registry homogenization; the images > were previously team-scoped as `team-…/terminal-lego:<task-id>-<build-hash>`). > The root `terminal-lego-task-image-map.jsonl` records the original build > provenance under the legacy refs and is retained for reproducibility, and the > original `environment/Dockerfile` remains for inspection and provenance. > > The exact exclusion records are stored in > [`prime-data-excluded-tasks.jsonl`](prime-data-excluded-tasks.jsonl). > > Exclusion reasons: > - `malformed_harbor_task`: 1 > - `no_op_pass`: 234 > - `no_op_validation_timeout`: 3 > - `oracle_gold_apply_failed`: 520 > - `oracle_test_failed`: 116 > - `oracle_validation_flaky`: 8 > - `oracle_validation_sandbox_error`: 4 > - `oracle_validation_timeout`: 96 > - `prime_image_build_timeout`: 242 --- # Terminal-Lego-15k: Docker-Verified Terminal Agent Tasks from Real StackOverflow Issues [Project Page](https://stephen0808.github.io/terminal-lego.github.io/) · [Code](https://github.com/SWE-Lego/terminal-lego) · [Paper](https://arxiv.org/pdf/2606.03461v1) · [Models](https://huggingface.co/SWE-Lego) **Terminal-Lego-15k** is a large-scale collection of **Docker-verified, Terminal-Bench-style agentic tasks** built from real StackOverflow technical issues. Each task is constructed through the [Terminal-Lego pipeline](https://github.com/SWE-Lego/terminal-lego): StackOverflow questions are filtered, converted via cascaded LLM generation, and retained only after Docker round-trip verification. This dataset is the task substrate behind the paper *What Makes Interaction Trajectories Effective for Training Terminal Agents?* It supports trajectory collection, SFT, and evaluation on [Terminal-Bench 2.0](https://github.com/laude-institute/terminal-bench) with the fixed **Terminus-2** harness. --- ## 🚀 Key Results & Performance Training student models on **15.3K Terminal-Lego trajectories** (collected from these tasks) yields strong gains on **Terminal-Bench 2.0** with exceptional data efficiency: | Model Size | Backbone (Qwen3) | Terminal-Lego SFT | Improvement | |------------|------------------|-------------------|-------------| | **8B** | 2.5% | **11.8%** | +9.3 pp | | **32B** | 3.4% | **24.3%** | +20.9 pp (~7×) | Compared with prior terminal-agent datasets at similar model scales: | Model | Size | Post-Training Data | Scaffold | TB 2.0 | |-------|------|--------------------|----------|--------| | Nemotron-Terminal-Qwen3-8B | 8B | 490.5K | Terminus-2 | **13.0%** | | **Terminal-Lego-Qwen3-8B** | 8B | **15.3K** | Terminus-2 | 11.8% | | Nemotron-Terminal-Qwen3-32B | 32B | 490.5K | Terminus-2 | **27.4%** | | **Terminal-Lego-Qwen3-32B** | 32B | **15.3K** | Terminus-2 | 24.3% | Terminal-Lego achieves competitive performance with **~30× less trajectory data** than large-scale baselines, while using real-world StackOverflow-grounded tasks rather than purely synthetic skill templates. --- ## 📂 Dataset Composition ### Scale & Verification | Statistic | Value | |-----------|-------| | Verified tasks | **15K+** | | Source domains | **90+** technical domains | | High-level categories | **13** domain groups | | Construction pipeline | StackOverflow → cascaded generation → test review → Docker verification | | Validation pass rate | **41.8%** (15,389 / 36,846 candidates) | ### Domain Distribution Tasks are sourced from diverse real-world terminal scenarios across 13 high-level categories (e.g., Core Terminal & OS, Python Ecosystem, Networking, Databases, ML & Data, Debugging, etc.): <p align="center"> <img src="assets/terminal_lego_source_domains_pie.png" alt="Terminal-Lego source domain distribution" width="720"/> </p> <p align="center"><em>Source domain distribution of Terminal-Lego tasks (90+ StackOverflow-grounded technical domains).</em></p> Representative domain tags include: `linux`, `bash`, `docker`, `git`, `python`, `pytorch`, `pandas`, `networking`, `ssh`, `nginx`, `sqlite`, `redis`, `regex`, `ffmpeg`, and more. ### What Each Task Contains Every task directory follows the **Harbor / Terminal-Bench 2.0** layout and is fully self-contained: ``` task_XXXXX/ ├── instruction.md # Task specification and success criteria ├── task.toml # Metadata (difficulty, category, tags, source_url, ...) ├── environment/ │ └── Dockerfile # Container environment definition ├── solution/ │ └── solve.sh # Reference oracle solution └── tests/ ├── test.sh └── test_outputs.py # Post-solution verification tests ``` **Pipeline stages (construction):** 1. **Source Collection** — Sample StackOverflow questions from 90+ domains with accepted answers and vote-based filtering. 2. **Cascaded Task Construction** — Dependent generation of instruction, environment, solution, Dockerfile, and tests. - **Stage 2.5 — Test Validation Loop** — `ast.parse()` syntax checks + independent LLM review (up to 3 rounds). 3. **Docker Round-Trip Verification** — Build image, run solution, execute tests; keep only tasks with positive post-solution reward. --- ## 📥 Usage ### Download a single task ```python from huggingface_hub import hf_hub_download path = hf_hub_download( repo_id="SWE-Lego/Terminal-Lego-15k", repo_type="dataset", filename="task_00000/instruction.md", ) ``` ### Clone the full dataset ```bash git lfs install git clone https://huggingface.co/datasets/SWE-Lego/Terminal-Lego-15k ``` ### Run with Harbor / Terminal-Bench Each `task_*` folder can be registered as a Harbor benchmark task. See the [Terminal-Lego code repository](https://github.com/SWE-Lego/terminal-lego) for the full data-generation and validation pipeline. ```bash # Example: inspect task metadata cat Terminal-Lego-15k/task_00000/task.toml ``` --- ## 🔗 Related Resources | Resource | Link | |----------|------| | Project page | https://stephen0808.github.io/terminal-lego.github.io/ | | Code & pipeline | https://github.com/SWE-Lego/terminal-lego | | Paper (arXiv) | https://arxiv.org/abs/2606.03461 | | SFT trajectories & models | [SWE-Lego org on Hugging Face](https://huggingface.co/SWE-Lego) | --- ## 📜 Citation If you use this dataset in your research, please cite: ```bibtex @misc{yang2026terminallego, title={What Makes Interaction Trajectories Effective for Training Terminal Agents?}, author={Sidi Yang and Chaofan Tao and Jierun Chen and Tiezheng Yu and Ruoyu Wang and Yuxin Jiang and Yiming Du and Wendong Xu and Jing Xiong and Taiqiang Wu and Lifeng Shang and Xiaohui Li and Ngai Wong and Haoli Bai}, year={2026}, eprint={2606.03461}, archivePrefix={arXiv}, primaryClass={cs.AI}, url={https://arxiv.org/abs/2606.03461} } ```