AlienKevin/SWE-ZERO-100K-Qwen3-1.7B-Base-ECHO-eval

---
license: apache-2.0
language:
- en
tags:
- swe-bench
- swe-zero
- eval
- mini-swe-agent
- qwen3
- echo
- terminalworld
size_categories:
- n<1K
---

# SWE-ZERO-100K-Qwen3-1.7B-Base-ECHO — SWE-bench Verified eval

Evaluation of a Qwen3-1.7B-Base SFT checkpoint trained on **100K** SWE-ZERO trajectories with **full-transcript loss** (the SFT-time analogue of [ECHO](https://github.com/SungSeongWoo/ECHO)-style unmasking — user/tool tokens count toward loss, not just assistant tokens), on the 100-task SWE-bench Verified slice from [marin#4898](https://github.com/marin-community/marin/issues/4898).

Scale-up of [AlienKevin/SWE-ZERO-10K-Qwen3-1.7B-Base-ECHO-eval](https://huggingface.co/datasets/AlienKevin/SWE-ZERO-10K-Qwen3-1.7B-Base-ECHO-eval) (the same loss-mask change at 10K). Companion to the assistant-only-masked baseline at the same scale (arm "a" 100K, pass@1 = 9/100 per [marin#5611](https://github.com/marin-community/marin/issues/5611)).

## Headline

**pass@1 = 7/100 = 7%** (latest trial per task)

For reference, the full 10K → 100K scaling story:

| arm | loss mask | 10K pass@1 | **100K pass@1** | Δ from 10× scaling |
|---|---|---:|---:|---:|
| (a) assistant-only | only assistant tokens count | 7 / 100 | **9 / 100** | +2 pp |
| (b) full transcript (this dataset) | + user/tool tokens count | 3 / 100 | **7 / 100** | +4 pp |
| **(a) − (b) gap** | — | **4 pp** | **2 pp** | gap halved |

The full-transcript regression at 10K (4 pp deficit) **halves to 2 pp at 100K**. Arm (b) scales faster (+4 pp vs +2 pp) — partial vindication of the "loss-budget dilution" hypothesis: when the action-token loss share drops to ~32%, the policy just needs more samples to recover the missing gradient signal per action token. The residual 2 pp gap at 100K is consistent with some env-token contamination on top of dilution, but it's small.

The published [ECHO](https://github.com/SungSeongWoo/ECHO) gains are **RL-time** (`L_GRPO + λ·L_env`); this dataset shows that the SFT-time analogue is *not categorically broken*, just data-hungry.

| pass@1 resolved task (latest trial reward=1) |
|---|
| django__django-12050 |
| django__django-13109 |
| django__django-13363 |
| django__django-15277 |
| django__django-15467 |
| pydata__xarray-4629 |
| pytest-dev__pytest-10081 |

Notably, (b) 100K **recovers three django solves it lost at 10K** (12050, 13109, 15467 — the "starved policy" tasks where 10K (b) wouldn't commit to an edit) plus picks up two new django solves and a new pytest. pass@N = pass@1 = 7/100 (no extra attempts unlocked anything).

## Model

- **GCS checkpoint:** `gs://marin-us-east5/checkpoints/exp5611_sft_qwen3_1_7b_swe_zero_100k_8192tokens_arch32k_echo_v5p-3beb0f/hf/step-6249`

## Setup

- **Base model:** `Qwen/Qwen3-1.7B-Base`
- **SFT data:** identical to arm (a) 100K — 100K random sample from [AlienKevin/SWE-ZERO-12M-trajectories @ 2f328e1d](https://huggingface.co/datasets/AlienKevin/SWE-ZERO-12M-trajectories/tree/2f328e1dcea8286aa8eb67ff5ec80c7fd4c99450), right-truncated to 8K tokens
- **Loss mask:** custom Qwen3 chat template wraps every user/tool message in `{% generation %}` so all tokens (not just assistant turns) contribute to the SFT loss
- **SFT compute:** v5p-16 us-east5, batch=16, 6,250 steps (1 epoch), LR 2e-5, cosine schedule (warmup 0.03, min_lr_ratio 0.1, wd 0.1, max_grad_norm 30) — byte-identical to arm (a) 100K except for the chat template
- **Eval framework:** Harbor + mini-swe-agent v1 + Daytona, same as 10K eval
- **Eval engine:** vLLM `max_model_len=32768`, temperature=1.0
- **Agent loop:** `max_turns=50`, `max_output_tokens=4096`, observation truncation at 10K chars
- **eos_token_id patched** to `[151643, 151645]` so vLLM stops at `<|im_end|>` (Qwen3 chat-template turn boundary)

## Coverage caveat

Of 100 tasks, 82 received at least one attempt with a usable (trajectory, reward) pair (vs 79 for arm (b) 10K, 93 for arm (a) 10K). 18 tasks had Daytona-side setup failures before the agent could run; they are recorded with `not_attempted: True` for transparency. This is a fixed cost of running on Daytona at the 100-instance cap, not a recipe issue.

## Schema

Identical to [arm (b) 10K](https://huggingface.co/datasets/AlienKevin/SWE-ZERO-10K-Qwen3-1.7B-Base-ECHO-eval):

- `task_id` (str): SWE-bench task ID
- `resolved` (bool): pass@1 — latest trial's reward > 0
- `resolved_any_attempt` (bool): pass@N — any attempt's reward > 0
- `reward` (float): latest attempt's reward
- `n_attempts` (int): number of trial attempts
- `n_assistant_turns` (int): assistant turns in the latest-attempt trajectory
- `trajectory` (list[dict]): full message list from the latest attempt
- `not_attempted` (bool, optional): True for tasks Daytona couldn't set up

## Tracking

- [marin-community/marin #5611](https://github.com/marin-community/marin/issues/5611) — SWE-ZERO Qwen3-1.7B eval recipe
- [marin-community/marin #5866](https://github.com/marin-community/marin/issues/5866) — TerminalWorld project
- [terminalworld.vercel.app](https://terminalworld.vercel.app) — project notes & the ongoing comparison