Open-AgentRL-SFT-3K

<div align="center">

<h1>Demystifying Reinforcement Learning in Agentic Reasoning<h1>

<p align="center"> <a href="https://arxiv.org/abs/2510.11701"> <img src="https://img.shields.io/badge/Paper-Arxiv-red?logo=arxiv&logoColor=red" alt="Paper on arXiv"/> </a> <a href="https://github.com/Gen-Verse/Open-AgentRL"> <img src="https://img.shields.io/badge/Open--AgentRL-GitHub-black?logo=github&logoColor=white" alt="Open-AgentRL on GitHub"/> </a> <a href="https://huggingface.co/datasets/Gen-Verse/Open-AgentRL-30K"> <img src="https://img.shields.io/badge/30K_RL_Dataset-Hugging%20Face-orange?logo=huggingface&logoColor=yellow" alt="30K RL Dataset"/> </a> <a href="https://huggingface.co/Gen-Verse/DemyAgent-4B"> <img src="https://img.shields.io/badge/DemyAgent--4B-Hugging%20Face-FFCC00?logo=huggingface&logoColor=yellow" alt="DemyAgent-4B Model"/> </a> </p> </div>

## 🎯 About This Repository
This repository contains the **3K Agentic SFT data**, which is our-self curated real, end-to-end multi-turn agentic reasoning dataset, consists problems from [s1k](https://huggingface.co/datasets/simplescaling/s1K-1.1) , [ReTool-SFT](https://huggingface.co/datasets/swordfaith/ReTool-SFT-multi-turn) and our self-curated LeetCode problems.
## 🌟 Introduction
In our work, we systematically investigate three dimensions of agentic RL: **data, algorithms, and reasoning modes**. Our findings reveal
- 🎯 **Data Quality Matters**: Real end-to-end trajectories and high-diversity datasets significantly outperform synthetic alternatives
- ⚡ **Training Efficiency**: Exploration-friendly techniques like reward clipping and entropy maintenance boost training efficiency
- 🧠 **Reasoning Strategy**: Deliberative reasoning with selective tool calls surpasses frequent invocation or verbose self-reasoning
We contribute high-quality SFT and RL datasets, demonstrating that **simple recipes enable even 4B models to outperform 32B models** on the most challenging reasoning benchmarks.
## 📦 Resources
| **Type** | **Name** | **Link** |
| --------- | ----------------------- | ------------------------------------------------------------ |
| 📊 Dataset | **3K Agentic SFT Data** | [🤗 HuggingFace](https://huggingface.co/datasets/Gen-Verse/Open-AgentRL-SFT-3K) |
| 📊 Dataset | 30K Agentic RL Data | [🤗 HuggingFace](https://huggingface.co/datasets/Gen-Verse/Open-AgentRL-30K) |
| 🤖 Model | Qwen2.5-7B-RA-SFT | [🤗 HuggingFace](https://huggingface.co/Gen-Verse/Qwen2.5-7B-RA-SFT) |
| 🤖 Model | Qwen3-4B-RA-SFT | [🤗 HuggingFace](https://huggingface.co/Gen-Verse/Qwen3-4B-RA-SFT) |
| 🤖 Model | DemyAgent-4B | [🤗 HuggingFace](https://huggingface.co/Gen-Verse/DemyAgent-4B) |
## 📝 Citation
```bibtex
@article{yu2025demystify,
title={Demystifying Reinforcement Learning in Agentic Reasoning},
author={Yu, Zhaochen and Yang, Ling and Zou, Jiaru and Yan, Shuicheng and Wang, Mengdi},
journal={arXiv preprint arXiv:2510.11701},
year={2025}
}
```

<div align="center">

<h1>揭秘智能体推理中的强化学习</h1>

<p align="center"> <a href="https://arxiv.org/abs/2510.11701"> <img src="https://img.shields.io/badge/Paper-Arxiv-red?logo=arxiv&logoColor=red" alt="arXiv 论文"/> </a> <a href="https://github.com/Gen-Verse/Open-AgentRL"> <img src="https://img.shields.io/badge/Open--AgentRL-GitHub-black?logo=github&logoColor=white" alt="GitHub 开源仓库 Open-AgentRL"/> </a> <a href="https://huggingface.co/datasets/Gen-Verse/Open-AgentRL-30K"> <img src="https://img.shields.io/badge/30K_RL_Dataset-Hugging%20Face-orange?logo=huggingface&logoColor=yellow" alt="Hugging Face 30K 强化学习数据集"/> </a> <a href="https://huggingface.co/Gen-Verse/DemyAgent-4B"> <img src="https://img.shields.io/badge/DemyAgent--4B-Hugging%20Face-FFCC00?logo=huggingface&logoColor=yellow" alt="Hugging Face DemyAgent-4B 模型"/> </a> </p> </div>

## 🎯 项目简介
本仓库包含**3K智能体监督微调（Supervised Fine-Tuning, SFT）数据集**，该数据集为我们精心整理的真实端到端多轮智能体推理数据集，其问题来源包括[s1k](https://huggingface.co/datasets/simplescaling/s1K-1.1)、[ReTool-SFT](https://huggingface.co/datasets/swordfaith/ReTool-SFT-multi-turn) 以及我们自行整理的LeetCode编程题。

## 🌟 研究概述
本研究系统探究了智能体强化学习（Agentic RL）的三大维度：**数据、算法与推理模式**，我们的核心发现如下：
- 🎯 **数据质量至关重要**：真实端到端交互轨迹与高多样性数据集的表现显著优于合成数据集
- ⚡ **训练效率优化**：奖励裁剪、熵维护等利于探索的技术可有效提升训练效率
- 🧠 **推理策略优化**：带有选择性工具调用的审慎推理，相较于频繁调用工具或冗长的自我推理，表现更优

我们开源了高质量的监督微调与强化学习数据集，实验证明**仅通过简单的训练范式，即可让4B参数模型在极具挑战性的推理基准测试中超越32B参数模型**。

## 📦 开源资源
| **资源类型** | **资源名称** | **下载链接** |
| --------- | ----------------------- | ------------------------------------------------------------ |
| 📊 数据集 | **3K智能体监督微调数据集** | [🤗 Hugging Face](https://huggingface.co/datasets/Gen-Verse/Open-AgentRL-SFT-3K) |
| 📊 数据集 | 30K智能体强化学习数据集 | [🤗 Hugging Face](https://huggingface.co/datasets/Gen-Verse/Open-AgentRL-30K) |
| 🤖 模型 | Qwen2.5-7B-RA-SFT | [🤗 Hugging Face](https://huggingface.co/Gen-Verse/Qwen2.5-7B-RA-SFT) |
| 🤖 模型 | Qwen3-4B-RA-SFT | [🤗 Hugging Face](https://huggingface.co/Gen-Verse/Qwen3-4B-RA-SFT) |
| 🤖 模型 | DemyAgent-4B | [🤗 Hugging Face](https://huggingface.co/Gen-Verse/DemyAgent-4B) |

## 📝 引用格式
bibtex
@article{yu2025demystify,
title={Demystifying Reinforcement Learning in Agentic Reasoning},
author={Yu, Zhaochen and Yang, Ling and Zou, Jiaru and Yan, Shuicheng and Wang, Mengdi},
journal={arXiv preprint arXiv:2510.11701},
year={2025}
}