SWE-ZERO-12M-trajectories

# SWE-ZERO 12M Trajectories The largest agentic-coding trace dataset to date: **112 B tokens** of execution-free agentic trajectories covering **122 K pull requests**, **3 K repositories**, and **16 programming languages**. ## Motivation Agentic mid-training has become a standard ingredient for frontier coding models: - **Code World Model** (Sept 2025) mid-trains on 5 T tokens, using 3 M agentic trajectories from 10.2 K Docker images across 3.15 K repos. - **Kimi-Dev** (Dec 2025) mid-trains on ~150 B tokens of "high-quality and real-world data" to instill BugFixer and TestWriter priors. - **DeepSeek-V4** (May 2026) incorporates agentic data during mid-training to enhance coding capability. All three rely on **containerized execution** to verify trajectories, and that is where the data ceiling sits. Kimi-Dev was first to call it out: > "while repository snapshots from GitHub are available, not all snapshots are equipped with an executable Docker environment." The current largest containerized collection, **SWE-rebench-V2**, ships 32 K tasks with pre-built images, and concedes the ceiling by additionally releasing **120 K+ tasks that could not be containerized** (4× more) with only install instructions and fail-to-pass metadata, because building those images is prohibitively expensive. **SWE-ZERO** (prior work) was the first to propose an *execution-free* pipeline for agentic SWE training data, framing the two Docker-imposed bottlenecks: - **Data scalability**: a large fraction of real-world repositories and PRs are discarded because intricate or legacy build configurations do not work in modern containerized environments. - **Training scalability**: orchestrating thousands of task-specific Docker images creates massive infrastructure overhead that complicates large-scale optimization and RL. SWE-ZERO sidesteps the Docker requirement by instructing the model not to run repo-specific commands like `pytest`, `npm test`, or build scripts that depend on the project's installed environment. The model is still free to explore and edit the repository using standard POSIX commands that need no setup: `grep`, `find`, `cat`, `sed`, `ls`, `git`, and so on. None of those require the repo to be built or its dependencies installed, so the pipeline can run against any GitHub PR snapshot and unlocks the long tail of PRs that prior pipelines must throw away. SWE-ZERO released **9 B tokens**, and we scale this **10× to 112 B tokens** using the same execution-free recipe. Coverage: - **12,290,800** rollouts - **122,908** unique PRs - **3,222** repositories - **16** programming languages To our knowledge it is now **5.7× larger than the previous largest agent-trace dataset, `open-thoughts/AgentTrove`** (111.06 B vs 19.41 B tokens): | Dataset | Release | Rollouts | Total tokens | |---|---|---:|---:| | `ricdomolm/mini-coder-trajs-400k` | 2025-09-30 | 396,515 | 5.26 B | | `nvidia/Nemotron-Terminal-Corpus` | 2026-02-19 | 366,154 | 6.23 B | | `nvidia/SWE-Zero-openhands-trajectories` (original SWE-ZERO release) | 2026-04-17 | 318,115 | 8.92 B | | `open-thoughts/AgentTrove` | 2026-04-27 | 1,696,847 | 19.41 B | | **`AlienKevin/SWE-ZERO-12M-trajectories` (this dataset, 10× scale-up)** | **2026-05-12** | **12,290,800** | **111.06 B** | ## Experiment validation **Diversity check (MinHash-64).** We follow the diversity protocol from the Code World Model paper. It measures the mean Jaccard similarity between rollouts in the same PR, on bash-command shingles. Anything under 0.5 is considered diverse. On our corpus, we measure **0.2730**, well under the threshold. **Training check.** To verify the trajectories actually transfer into model capability, we trained Marin-8B base on three subsets of an earlier snapshot of this dataset ([`@ab6295c3`](https://huggingface.co/datasets/AlienKevin/SWE-ZERO-12M-trajectories/tree/ab6295c37153e938c871d4021c4457379e67396e), 1.6 M rollouts / 18.4 B tokens) and evaluated on **SWE-bench Verified** (100 random tasks, mini-swe-agent v1 + Harbor, 3 runs per model, temperature 1.0): | Training data | Rollouts | Tokens | Mean resolve rate | |---|---:|---:|---:| | Marin-8B Base | n/a | n/a | **0.0 %** | | SWE-ZERO training | 10,000 | ~100 M | **3.3 % ± 0.6** | | SWE-ZERO training | 50,000 | ~570 M | **4.0 % ± 2.0** | | SWE-ZERO training | 100,000 | ~1.1 B | **5.3 % ± 1.5** |  Resolve rate scales consistently with training data through the 1 B-token point we tested. For methodological simplicity these runs use the standard SFT masked-loss recipe, but this corpus is intended as a **mid-training dataset**. Downstream models should be further post-trained on dedicated SFT datasets such as [`open-thoughts/AgentTrove`](https://huggingface.co/datasets/open-thoughts/AgentTrove), [`nvidia/SWE-Hero-openhands-trajectories`](https://huggingface.co/datasets/nvidia/SWE-Hero-openhands-trajectories), and [`ricdomolm/mini-coder-trajs-400k`](https://huggingface.co/datasets/ricdomolm/mini-coder-trajs-400k). Full methodology, per-task breakdown, and other experiment details: [marin-community/marin#4898 (comment)](https://github.com/marin-community/marin/issues/4898#issuecomment-4283675417). ## Methodology Each rollout starts from a real GitHub PR snapshot in [`nebius/SWE-rebench-V2-PRs`](https://huggingface.co/datasets/nebius/SWE-rebench-V2-PRs), which provides real-world software-engineering tasks spanning 3,222 repos. We sample **100 independent multi-turn rollouts per PR** from [`ricdomolm/mini-coder-1.7b`](https://huggingface.co/ricdomolm/mini-coder-1.7b) (a compact 1.7 B-parameter code model that scores 50.4 pass@100 on SWE-bench Verified) at temperature 1.0. Trajectories follow the [`mini-swe-agent` v1](https://github.com/SWE-agent/mini-SWE-agent) format: a `messages` list where the assistant issues one bash command per turn and the harness returns the command output. Following SWE-ZERO, generation is fully **execution-free**: no container is built, no test is executed, no verifier is consulted. That is what unlocks the scale. ## Schema | field | type | |-------|------| | `instance_id` | string - PR identifier | | `repo` | string - `owner/name` | | `messages` | list[{role, content}] - multi-turn agentic trajectory | | `trajectory_format` | string - `mini-swe-agent-1` | | `exit_status` | string - `Submitted`, `incomplete`, etc. | | `duration_sec` | float64 - wall-clock generation time | ## Sampling settings | Setting | Value | |---------|-------| | Max turns per rollout | 15 | | Rollouts per PR | 100 | | Sampling temperature | 1.0 | | Max model length (vLLM) | 32,768 tokens | | Max sequences (vLLM) | 256 | | Max total tokens per rollout | 32,768 | ## Limitations This is a **mid-training corpus**, not an SFT dataset. Treat it accordingly. - **Skill acquisition, not task completion.** The goal of this corpus is to instill **agentic tool-use priors** in a base model at scale: how to drive a bash shell, navigate a repository, read and edit files, and interleave thinking with actions. It is not designed to teach the model to *solve* SWE-bench-style tasks end-to-end. That is what downstream RL and SFT stages are for. - **No verification.** Because SWE-ZERO is execution-free, no trajectory has been checked against ground-truth tests. Patches are not known to compile, apply cleanly, or pass. Correctness was deliberately traded for scale and repo coverage. - **Low submission rate.** Most rollouts do not reach a successful `Submitted` exit status; many exit `incomplete` or hit other terminal conditions. If you filter to clean submissions only, you will lose the bulk of the corpus. - **15-turn cap.** Trajectories are truncated at 15 model turns to keep p99 token cost bounded and throughput high. Many tasks are not solvable in 15 turns and the trajectory simply ends mid-investigation. That said, prior work shows these limitations are forgiving in practice. Both **SERA** and **NemotronTerminal** report that downstream SFT performance is robust to incomplete, failed, and length-truncated traces. Concretely, NemotronTerminal's Table 7 shows that **no filtering (12.4 %)** significantly surpasses both **complete-only (6.74 %)** and **success-only (5.06 %)** filtering strategies. Our own experiments with Qwen3-1.7-Base corroborate this: the model reaches **9 %** on SWE-bench Verified when trained on 100K samples from this corpus truncated to 8K tokens, even though most rollouts do not end with a patch submission. Full details in [marin-community/marin#5611 (comment)](https://github.com/marin-community/marin/issues/5611#issuecomment-4440640008). ## Acknowledgements Thanks to [TPU Research Cloud](https://sites.research.google/trc/about/) for sponsoring all the compute for this project! ## Citation ```bibtex @misc{li2026swezero12m, author = {Xiang Li}, title = {{SWE-ZERO-12M-trajectories: 112B Tokens of Execution-Free Agentic Trajectories}}, year = {2026}, publisher = {Hugging Face}, howpublished = {\url{https://huggingface.co/datasets/AlienKevin/SWE-ZERO-12M-trajectories}}, } ```