acebench

# ACEBench Dataset This repository contains the [ACEBench](https://github.com/ACEBench/ACEBench/tree/main/data_all/data_en) dataset, formatted for evaluating and training tool-using language models. The dataset has been processed into a unified structure, with problem descriptions merged with their corresponding ground-truth rubrics. Notebook used to format the dataset: [Open in Colab](https://colab.research.google.com/drive/1p1ktldFIlekxclmoDopw_kBQm80uTwAi#revisionId=0B7zJoz05LiPWcVlQMHJqTkFUbCtpQkNLaExaSGVvc21HSzVNPQ) ## Dataset Structure The dataset is provided under a single configuration, **`en`**, which contains three distinct **splits**: - **`normal`**: Standard tool-use scenarios. This is the largest set and covers a wide range of common use cases. - **`agent`**: Complex, multi-step tasks that require agentic reasoning, planning, and sequential tool use. - **`special`**: Scenarios designed to test the model's robustness, including handling irrelevant information, incomplete instructions, or error-prone tool parameters. ## How to Use You can load any of the splits from the `en` configuration using the `datasets` library. ```python from datasets import load_dataset # Load the 'normal' split from the 'en' configuration normal_data = load_dataset("oliveirabruno01/acebench", "en", split="normal") # Load the 'agent' split agent_data = load_dataset("oliveirabruno01/acebench", "en", split="agent") # Load the 'special' split special_data = load_dataset("oliveirabruno01/acebench", "en", split="special") print(agent_data[0]) ``` ## Data Fields Each data point in every split has the following fields. All fields are stored as strings, with complex objects serialized as JSON strings. - `id` (string): A unique identifier for the data point. - `question` (string): The user's prompt or the problem statement for the agent to solve. - `sub_category` (string): The original filename/sub-category of the data point (e.g., `data_normal_single_turn_single_function`). - `function` (string): A JSON string representing the list of available tools (functions) for the agent. - `rubric` (string): A JSON string containing the ground truth for verification. This may include the final answer or the "golden path" of tool calls. - *Other fields*: The dataset also includes other metadata from the original files like `initial_config`, `profile`, `time`, etc., all cast to strings. ## Citation If you use this dataset, please cite the original ACEBench paper: ```bibtex @article{chen2025acebench, title={ACEBench: Who Wins the Match Point in Tool Learning?}, author={Chen, Chen and Hao, Xinlong and Liu, Weiwen and Huang, Xu and Zeng, Xingshan and Yu, Shuai and Li, Dexun and Wang, Shuai and Gan, Weinan and Huang, Yuefeng and others}, journal={arXiv preprint arXiv:2501.12851}, year={2025} } ```