AlienKevin/SWE-ZERO-1M-Qwen3-1.7B-Base-ECHO-eval

该数据集记录了在SWE-bench Verified的100个任务上，对一个基于Qwen3-1.7B-Base模型、使用100万条SWE-ZERO轨迹并应用全转录损失（full-transcript loss）进行监督微调（SFT）的检查点所进行的评估结果。评估旨在研究在扩大训练数据规模（从10K、100K到1M）时，使用全转录损失（即训练时用户和工具令牌也计入损失）与仅使用助手令牌损失相比，对模型在软件工程任务（如代码修复）上性能的影响。数据集包含每个任务的评估结果，如任务是否解决（resolved）、奖励值（reward）、尝试次数（n_attempts）、助手交互轮次（n_assistant_turns）以及完整的交互轨迹（trajectory）。结果显示，在1M规模下，全转录损失模型的pass@1得分为4%（4/100），与基线模型（11/100）相比存在7个百分点的性能差距，且趋势与较小规模时相反。分析表明模型行为健康，未出现病理性问题，但性能差距可能源于统计噪声或训练配置差异。

This dataset contains the evaluation results of a Qwen3-1.7B-Base model checkpoint that was supervised fine-tuned (SFT) on 1 million SWE-ZERO trajectories using a full-transcript loss (where user and tool tokens contribute to the loss, not just assistant tokens, analogous to ECHO-style unmasking). The evaluation is performed on the 100-task SWE-bench Verified slice. The dataset aims to study the impact of scaling training data (from 10K, 100K to 1M) and using full-transcript loss versus assistant-only loss on model performance in software engineering tasks (e.g., code repair). It includes per-task evaluation metrics such as whether the task was resolved (resolved), reward value, number of attempts, number of assistant turns, and the full interaction trajectory. Results show that at the 1M scale, the full-transcript loss model achieved a pass@1 score of 4% (4/100), exhibiting a 7 percentage-point gap compared to the baseline model (11/100), with a trend reversal from smaller scales. Analysis indicates healthy model behavior without pathological issues, but the performance gap may be attributed to statistical noise or training configuration differences.