txchmechanicus/CoderForge-Preview

--- dataset_info: - config_name: trajectories features: - name: trajectory_id dtype: string - name: finish_reason dtype: string - name: image dtype: string - name: messages dtype: string - name: reward dtype: float64 - name: tools dtype: string - name: license dtype: string splits: - name: SWE_Rebench num_bytes: 19392208677 num_examples: 77169 - name: SWE_Smith num_bytes: 33088967556 num_examples: 148001 - name: R2E_Gym num_bytes: 6869123922 num_examples: 32964 - name: filtered_reward1 num_bytes: 33547502194 num_examples: 155144 download_size: 22788997561 dataset_size: 92897802349 - config_name: trajectories-tokenized_qwencoder features: - name: trajectory_id dtype: string - name: reward dtype: float64 - name: chat_template_applied dtype: string - name: input_ids list: int32 - name: labels list: int64 splits: - name: SWE_Rebench num_bytes: 64238782798 num_examples: 77169 - name: SWE_Smith num_bytes: 107118447512 num_examples: 148001 - name: R2E_Gym num_bytes: 23869485518 num_examples: 32964 - name: filtered_reward1 num_bytes: 108349044091 num_examples: 155144 download_size: 49985669802 dataset_size: 303575759919 configs: - config_name: trajectories data_files: - split: SWE_Rebench path: trajectories/SWE_Rebench-* - split: SWE_Smith path: trajectories/SWE_Smith-* - split: R2E_Gym path: trajectories/R2E_Gym-* - split: filtered_reward1 path: trajectories/filtered_reward1-* - config_name: trajectories-tokenized_qwencoder data_files: - split: SWE_Rebench path: trajectories-tokenized_qwencoder/SWE_Rebench-* - split: SWE_Smith path: trajectories-tokenized_qwencoder/SWE_Smith-* - split: R2E_Gym path: trajectories-tokenized_qwencoder/R2E_Gym-* - split: filtered_reward1 path: trajectories-tokenized_qwencoder/filtered_reward1-* --- # CoderForge-Preview: SOTA Open Dataset for Training Efficient Agents **CoderForge-Preview** is **the** **largest open test-verified coding agent dataset.** Fine-tuning Qwen-3 32B on it, we boost **SWE-Bench Verified performance** **23.0% → 59.4% pass@1** and rank **#1 among open-data** and **#2 among open-weight models ≤32B parameters.**   ## Limitations - **Adaptability to different scaffolds:** We generated all trajectories using a **single scaffold** and **fixed tool set** (no permutations). Models trained via SFT on this data may perform worse when deployed with **different scaffolds, tools, prompt templates, or tool-call formats**. - **Task scope:** Our data sources skew toward **bug fixing**. As a result, models trained on this dataset may be less capable on tasks outside that scope, such as **feature implementation**, **refactors**, or **design-heavy changes**. - **User interaction:** Real coding-agent usage often involves **ongoing user collaboration**, with user messages appearing throughout the trajectory—not just at the start. This kind of interactive supervision is still largely missing from open coding-agent datasets (including ours). Models trained on SFT alone may therefore underperform in **interactive settings**. ## Conclusion In this release, we focus on **large-scale agentic data generation**: assembling **51K distinct open-source tasks** and generating **long-horizon, multi-step SFT trajectories**. Our results show that a simple data-generation pipeline combined with **pure SFT** can produce substantial gains in coding-agent performance. ### Next steps Moving forward, we plan to: - **Scale data generation further** (more tasks, more trajectories, longer horizons where helpful) - Generate data under **multiple scaffolds**, **tool sets**, and **prompt/tool-call permutations** to improve robustness and transfer - Train **larger models** and run more systematic **hyperparameter tuning** - Follow the **DeepSWE** training paradigm by applying **agentic reinforcement learning** on top of our fine-tuned model to drive further performance gains ## Citation ```bibtex @misc{CoderForge2026, title = {CoderForge-Preview: SOTA Open Dataset for Training Efficient Agents}, author = {Ariyak, Alpay and Zhang, Junda and Wang, Junxiong and Zhu, Shang and Bianchi, Federico and Srivastava, Sanjana and Panda, Ashwinee and Bharti, Siddhant and Xu, Chenfeng and Heo, John and Wu, Xiaoxia Shirley and Zhou, James and Liang, Percy and Song, Leon and Zhang, Ce and Athiwaratkun, Ben and Zhou, Zhongzhu and Wu, Qingyang}, year = {2026}, month = feb, publisher = {TogetherAI Blog}, url = {https://www.together.ai/blog/coderforge-preview}, note = {Project core leads: Alpay Ariyak; Zhongzhu Zhou; Qingyang Wu} } ```数据集信息： - 配置名称：trajectories（轨迹集） 特征： - 名称：trajectory_id（轨迹ID），数据类型：字符串（string） - 名称：finish_reason（终止原因），数据类型：字符串（string） - 名称：image（图像），数据类型：字符串（string） - 名称：messages（对话消息），数据类型：字符串（string） - 名称：reward（奖励值），数据类型：float64 - 名称：tools（工具集），数据类型：字符串（string） - 名称：license（许可证），数据类型：字符串（string） 划分集： - 名称：SWE_Rebench，字节数：19392208677，样本数：77169 - 名称：SWE_Smith，字节数：33088967556，样本数：148001 - 名称：R2E_Gym，字节数：6869123922，样本数：32964 - 名称：filtered_reward1，字节数：33547502194，样本数：155144 下载大小：22788997561，数据集总大小：92897802349 - 配置名称：trajectories-tokenized_qwencoder（经过Qwen编码器分词的轨迹集） 特征： - 名称：trajectory_id（轨迹ID），数据类型：字符串（string） - 名称：reward（奖励值），数据类型：float64 - 名称：chat_template_applied（已应用的对话模板），数据类型：字符串（string） - 名称：input_ids（输入Token序列），数据类型：int32列表（list: int32） - 名称：labels（标签序列），数据类型：int64列表（list: int64） 划分集： - 名称：SWE_Rebench，字节数：64238782798，样本数：77169 - 名称：SWE_Smith，字节数：107118447512，样本数：148001 - 名称：R2E_Gym，字节数：23869485518，样本数：32964 - 名称：filtered_reward1，字节数：108349044091，样本数：155144 下载大小：49985669802，数据集总大小：303575759919 配置项： - 配置名称：trajectories（轨迹集），数据文件： - 划分集：SWE_Rebench，路径：trajectories/SWE_Rebench-* - 划分集：SWE_Smith，路径：trajectories/SWE_Smith-* - 划分集：R2E_Gym，路径：trajectories/R2E_Gym-* - 划分集：filtered_reward1，路径：trajectories/filtered_reward1-* - 配置名称：trajectories-tokenized_qwencoder（经过Qwen编码器分词的轨迹集），数据文件： - 划分集：SWE_Rebench，路径：trajectories-tokenized_qwencoder/SWE_Rebench-* - 划分集：SWE_Smith，路径：trajectories-tokenized_qwencoder/SWE_Smith-* - 划分集：R2E_Gym，路径：trajectories-tokenized_qwencoder/R2E_Gym-* - 划分集：filtered_reward1，路径：trajectories-tokenized_qwencoder/filtered_reward1-* # CoderForge-Preview：用于训练高效智能体的前沿开放数据集 **CoderForge-Preview** 是目前规模最大的经过测试验证的开源代码智能体数据集。 在该数据集上对Qwen-3 32B模型进行微调后，我们将SWE-Bench验证集的pass@1指标从23.0%提升至59.4%，在开源数据驱动模型中排名第一，在参数量不超过32B的开源权重模型中排名第二。   ## 局限性 - **不同脚手架的适应性**：我们使用单一脚手架与固定工具集（无排列组合）生成了所有轨迹。基于该数据通过监督微调（SFT）训练的模型，在部署时若使用不同的脚手架、工具集、提示模板或工具调用格式，性能可能会下降。 - **任务范围偏向**：我们的数据源主要聚焦于缺陷修复任务。因此，在该数据集上训练的模型，在功能实现、代码重构或设计性变更等任务上的表现可能欠佳。 - **用户交互缺失**：实际的代码智能体使用场景通常涉及持续的用户协作，对话消息会贯穿整个轨迹过程，而非仅出现在开头。这类交互式监督在当前开源代码智能体数据集（包括本数据集）中仍较为稀缺。仅通过监督微调训练的模型，在交互式场景下的性能可能不尽如人意。 ## 结论 在本次发布中，我们聚焦于大规模智能体数据生成：整合了51K个不同的开源任务，并生成了长时序多步骤的监督微调轨迹。我们的实验结果表明，将简单的数据生成流程与纯监督微调相结合，能够显著提升代码智能体的性能。 ### 后续计划 我们后续将推进以下工作： - 进一步扩大数据生成规模（增加更多任务、轨迹，在合适场景下延长时序长度） - 在多种脚手架、工具集以及提示/工具调用排列组合下生成数据，以提升模型的鲁棒性与迁移能力 - 训练更大规模的模型，并开展更系统的超参数调优 - 遵循DeepSWE训练范式，在微调后的模型基础上应用智能体强化学习，以进一步提升模型性能 ## 引用 bibtex @misc{CoderForge2026, title = {CoderForge-Preview: SOTA Open Dataset for Training Efficient Agents}, author = {Ariyak, Alpay and Zhang, Junda and Wang, Junxiong and Zhu, Shang and Bianchi, Federico and Srivastava, Sanjana and Panda, Ashwinee and Bharti, Siddhant and Xu, Chenfeng and Heo, John and Wu, Xiaoxia Shirley and Zhou, James and Liang, Percy and Song, Leon and Zhang, Ce and Athiwaratkun, Ben and Zhou, Zhongzhu and Wu, Qingyang}, year = {2026}, month = feb, publisher = {TogetherAI Blog}, url = {https://www.together.ai/blog/coderforge-preview}, note = {Project core leads: Alpay Ariyak; Zhongzhu Zhou; Qingyang Wu} }