RioLee/ToolPref-Pairwise-30K
收藏资源简介：
--- license: cc-by-nc-sa-4.0 task_categories: - text-classification language: - en tags: - function-calling - LLM agent - reward modeling size_categories: - 10K<n<100K --- # ToolPref-Pairwise-30K <p align="left"> <a href="https://arxiv.org/abs/2510.26167">[Paper]</a> | <a href="https://huggingface.co/RioLee/ToolRM-Qwen3-4B-Thinking-2507">[Model]</a> | <a href="https://huggingface.co/datasets/RioLee/TRBench-BFCL">[Benchmark]</a> | <a href="https://github.com/lirenhao1997/ToolRM">[Code]</a> </p> ## 💡 Summary This dataset is a part of [One Model to Critique Them All: Rewarding Agentic Tool-Use via Efficient Reasoning](https://arxiv.org/abs/2510.26167). It comprises 30,000 preference annotations in agentic tool-use scenarios and was used to train the [ToolRM](https://huggingface.co/RioLee/ToolRM-Qwen3-4B-Thinking-2507) model series. ## 🌟 Overview ToolRM is a family of lightweight generative reward models tailored for general tool-use scenarios. To build these models, we propose a novel pipeline that constructs pairwise preference data using rule-based scoring and multidimensional sampling. This yields [ToolPref-Pairwise-30K](https://huggingface.co/datasets/RioLee/ToolPref-Pairwise-30K), a diverse, balanced, and challenging dataset of critique tasks that supports reinforcement learning with verifiable feedback. To evaluate tool-use RMs, we also introduce [TRBench-BFCL](https://huggingface.co/datasets/RioLee/TRBench-BFCL), a benchmark built on the agentic evaluation suite BFCL. Trained on our constructed data, models from the Qwen3-4B/8B series outperform several giant LLMs in pairwise reward judgments. Beyond training objectives, ToolRM generalizes to broader critique tasks, including Best-of-N sampling and self-correction. ## 🔧 Usage The datasets are saved as parquet files. You can easily load them with `datasets` library: ```python from datasets import load_dataset dataset = load_dataset('parquet', data_files=<your_local_dataset_path>) data_list = dataset['train'].to_list() ``` For each JSON-formatted data sample, you can either use the templated prompt in the `prompt` field for the critique task or use the original messages in `chat_history`, `chosen_response`, and `reject_response`, as you prefer. The ground truth answer of each sample is saved as a string in `reward_model`->`ground_truth`. Please refer to this [github repo](https://github.com/lirenhao1997/ToolRM) for guidelines on training generative reward models with this dataset. Note that we respectively use the `think` and `no_think` prompt templates to create the datasets. Upon training, use the `*_think` dataset for reasoning models and the `*_no_think` dataset for non-reasoning models. ## 🚦 Data Licenses ToolRM is a research project developed by Alibaba Cloud and licensed under the CC BY-NC-SA 4.0 License. A part of this dataset was generated using Gemini-2.5 / Claude-3.7 and should not be used to develop models that compete with Google / Anthropic. ## 🙏 Acknowledgments We are grateful to the following projects for contributing high-quality tool-use trajectories, which underpinned the construction of the ToolPref dataset: [APIGen](https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k), [APIGen-MT](https://huggingface.co/datasets/Salesforce/APIGen-MT-5k), [BUTTON](https://github.com/PKU-Baichuan-MLSystemLab/BUTTON/tree/main/data), [ComplexFuncBench](https://huggingface.co/datasets/zai-org/ComplexFuncBench), [Glaive-Function-Calling](https://huggingface.co/datasets/glaiveai/glaive-function-calling-v2), [Hermes-Function-Calling](https://huggingface.co/datasets/NousResearch/hermes-function-calling-v1), [ToolAlpaca](https://github.com/tangqiaoyu/ToolAlpaca/tree/main/data). ## 📝 Citation If you find our work helpful, feel free to give us a cite. ``` @misc{li2025modelcritiqueallrewarding, title={One Model to Critique Them All: Rewarding Agentic Tool-Use via Efficient Reasoning}, author={Renhao Li and Jianhong Tu and Yang Su and Hamid Alinejad-Rokny and Derek F. Wong and Junyang Lin and Min Yang}, year={2025}, eprint={2510.26167}, archivePrefix={arXiv}, primaryClass={cs.AI}, url={https://arxiv.org/abs/2510.26167}, } ```
The ToolPref-Pairwise-30K dataset consists of 30,000 preference annotations in agentic tool-use scenarios and is used to train the ToolRM model series. The dataset is constructed with rule-based scoring and multidimensional sampling to create pairwise preference data, forming a diverse, balanced, and challenging dataset of critique tasks that support reinforcement learning with verifiable feedback.
