RioLee/TRBench-BFCL
收藏资源简介：
--- license: apache-2.0 language: - en tags: - function-calling - LLM agent - reward model task_categories: - text-classification --- # TRBench-BFCL <p align="left"> <a href="https://arxiv.org/abs/2510.26167">[Paper]</a> | <a href="https://huggingface.co/RioLee/ToolRM-Qwen3-4B-Thinking-2507">[Model]</a> | <a href="https://huggingface.co/datasets/RioLee/TRBench-BFCL">[Benchmark]</a> | <a href="https://github.com/lirenhao1997/ToolRM">[Code]</a> </p> ## 💡 Summary This dataset is a part of [One Model to Critique Them All: Rewarding Agentic Tool-Use via Efficient Reasoning](https://arxiv.org/abs/2510.26167) and serves as **a dedicated benchmark for evaluating reward models in tool-use settings**. It comprises 2,983 preference annotations buit upon [BFCL V3](https://gorilla.cs.berkeley.edu/blogs/13_bfcl_v3_multi_turn.html), with assistant responses extracted from archived trajectories available in this [github repo](https://github.com/HuanzhiMao/BFCL-Result). ## 🌟 Overview ToolRM is a family of lightweight generative reward models tailored for general tool-use scenarios. To build these models, we propose a novel pipeline that constructs pairwise preference data using rule-based scoring and multidimensional sampling. This yields [ToolPref-Pairwise-30K](https://huggingface.co/datasets/RioLee/ToolPref-Pairwise-30K), a diverse, balanced, and challenging dataset of critique tasks that supports reinforcement learning with verifiable feedback. To evaluate tool-use RMs, we also introduce [TRBench-BFCL](https://huggingface.co/datasets/RioLee/TRBench-BFCL), a benchmark built on the agentic evaluation suite BFCL. Trained on our constructed data, models from the Qwen3-4B/8B series outperform several giant LLMs in pairwise reward judgments. Beyond training objectives, ToolRM generalizes to broader critique tasks, including Best-of-N sampling and self-correction. ## 🔧 Usage The datasets are saved as parquet files. You can easily load them with `datasets` library: ```python from datasets import load_dataset dataset = load_dataset('parquet', data_files=<your_local_dataset_path>) data_list = dataset['train'].to_list() ``` Each data sample is duplicated, with the response order swapped in the second half of the dataset, to minimize position bias in LLM evaluation. For each JSON-formatted data sample, you can either use the templated prompt in the `prompt` field for the critique task or use the original messages in `chat_history`, `chosen_response`, and `reject_response`, as you prefer. The ground truth answer of each sample is saved as a string in `reward_model`->`ground_truth`. Please refer to this [github repo](https://github.com/lirenhao1997/ToolRM) for evaluating different reward models on this benchmark. Note that we respectively use the `think` and `no_think` prompt templates to create the datasets. Upon evaluations, use the `*_think` dataset for reasoning models and the `*_no_think` dataset for non-reasoning models. ## 📊 Evaluation Results On TRBench-BFCL, we evaluate and report the weighted-average classification accuracy for the following reward models: | Models | W-Avg. | |--------|--------| | *--------Proprietary & Open-source Frontier LLMs--------* || | DeepSeek-AI/DeepSeek-R1-0528 | 57.93 | | OpenAI/GPT-4o-2024-11-20 | 59.00 | | OpenAI/o3-2025-04-16 | 59.40 | | Google/Gemini-2.5-Flash | 59.87 | | Google/Gemini-2.5-Pro | 59.94 | | Qwen/Qwen3-235B-A22B-Thinking-2507 | 60.64 | | DeepSeek-AI/DeepSeek-V3-0324 | 61.45 | | Qwen/Qwen2.5-Max | 62.39 | | Anthropic/Claude-3.7-Sonnet | 62.45 | | Anthropic/Claude-4-Sonnet | 64.23 | | *--------Open-source Reward Models--------* || | [Databricks/CLoud-RM-Llama-3-8B](https://huggingface.co/ankner/Llama3-8B-CLoud-RM) | 37.34 | | [Unbabel/M-Prometheus-7B](https://huggingface.co/Unbabel/M-Prometheus-7B) | 51.19 | | [Microsoft-Research/RRM-7B](https://huggingface.co/Reward-Reasoning/RRM-7B) | 56.05 | | [UIUC/RM-R1-DeepSeek-Distilled-Qwen-32B](https://huggingface.co/gaotang/RM-R1-DeepSeek-Distilled-Qwen-32B) | 56.25 | | [Unbabel/M-Prometheus-14B](https://huggingface.co/Unbabel/M-Prometheus-14B) | 56.32 | | [Skywork/Skywork-Critic-Llama-3.1-8B](https://huggingface.co/Skywork/Skywork-Critic-Llama-3.1-8B) | 56.92 | | [Skywork/Skywork-Critic-Llama-3.1-70B](https://huggingface.co/Skywork/Skywork-Critic-Llama-3.1-70B) | 60.31 | | [Microsoft-Research/RRM-32B](https://huggingface.co/Reward-Reasoning/RRM-32B) | 64.50 | | [Skywork/Skywork-Reward-Llama-3.1-8B-v0.2](https://huggingface.co/Skywork/Skywork-Reward-Llama-3.1-8B-v0.2) | 70.23 | | [InternLM/InternLM2-7B-Reward](https://huggingface.co/internlm/internlm2-7b-reward) | 71.17 | | *--------Models Trained on ToolPref-Pairwise-30K--------* || | [Qwen/Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507) | 59.67 | | ToolRM-Qwen3-4B-Instruct-2507 | 66.85 (+7.18) | | [Qwen/Qwen3-4B (Thinking mode)](https://huggingface.co/Qwen/Qwen3-4B) | 59.34 | | ToolRM-Qwen3-4B | 68.89 (+9.55) | | [Qwen/Qwen3-8B (Thinking mode)](https://huggingface.co/Qwen/Qwen3-8B) | 59.44 | | ToolRM-Qwen3-8B | 68.92 (+9.48) | | [Qwen/Qwen3-4B-Thinking-2507](https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507) | 57.59 | | **[ToolRM-Qwen3-4B-Thinking-2507](https://huggingface.co/RioLee/ToolRM-Qwen3-4B-Thinking-2507)** | **71.87 (+14.28)** | Please refer to our paper for more detailed evaluation results. ## 🙏 Acknowledgments We are grateful to [BFCL](https://github.com/ShishirPatil/gorilla/tree/main/berkeley-function-call-leaderboard) for providing a high-quality foundation for this reward-model benchmark, and to [fc-reward-bench](https://huggingface.co/datasets/ibm-research/fc-reward-bench) for pioneering evaluation in this field. ## 📝 Citation If you find our work helpful, feel free to give us a cite. ``` @misc{li2025modelcritiqueallrewarding, title={One Model to Critique Them All: Rewarding Agentic Tool-Use via Efficient Reasoning}, author={Renhao Li and Jianhong Tu and Yang Su and Hamid Alinejad-Rokny and Derek F. Wong and Junyang Lin and Min Yang}, year={2025}, eprint={2510.26167}, archivePrefix={arXiv}, primaryClass={cs.AI}, url={https://arxiv.org/abs/2510.26167}, } ```
TRBench-BFCL is a dedicated benchmark for evaluating reward models in tool-use settings, comprising 2,983 preference annotations based on BFCL V3.
