snorkelai/Tau2-Bench-Airline-With-Code-Agents
收藏资源简介：
--- dataset_info: features: - name: task_id dtype: string - name: model dtype: string - name: version dtype: string - name: user_scenario struct: - name: instructions struct: - name: domain dtype: string - name: known_info dtype: string - name: reason_for_call dtype: string - name: task_instructions dtype: string - name: unknown_info dtype: string - name: persona dtype: 'null' - name: db_update_required dtype: bool - name: trace list: - name: content dtype: string - name: role dtype: string - name: timestamp dtype: string - name: tool_calls dtype: string - name: turn_idx dtype: int64 - name: reward dtype: float64 - name: reward_breakdown struct: - name: COMMUNICATE dtype: float64 - name: DB dtype: float64 - name: db_diff dtype: string - name: db_diff_by_table struct: - name: flights dtype: string - name: reservations dtype: string - name: users dtype: string - name: db_diff_verbose dtype: string - name: __index_level_0__ dtype: int64 splits: - name: train num_bytes: 14972433 num_examples: 500 download_size: 4452463 dataset_size: 14972433 configs: - config_name: default data_files: - split: train path: data/train-* license: apache-2.0 task_categories: - text-generation language: - en tags: - agent - code --- # Dataset Card for a Code Agent Version of Tau Bench 2 Airline ## Dataset Summary This dataset includes sample traces and associated metadata from multi-turn interactions between an code agent and AI assistant. The dataset is based on the Airline environment from [Tau^2 Bench](https://github.com/sierra-research/tau2-bench) and contains traces from both the original version and a version made at Snorkel AI using code agents to solve the same tasks (indicator in the `version` field; details below). - **Curated by:** Snorkel AI - **License:** Apache-2.0 License Example snippet of a trace from Claude Sonnet 4.5 self-correcting after an exception: <img src="trace_eg1.png" style="width:50%;"> ## Quickstart To load the dataset, use the following code: ```python from datasets import load_dataset ds = load_dataset("snorkelai/Tau2-Bench-Airline-With-Code-Agents") ``` ## Dataset Results There is a lot of rich information in this dataset. The top-line result to highlight here is the differential effect of enabling code-generation on task reward depending on whether a database update is required. Models appear to do better when they are only challenged to make inferences and fail nearly every time for write operations. However, it is also worth noting that write operations are possible, which we see in a few examples with Claude Sonnet 4.5 and GPT-5 (as well as ad hoc testing we did). <img src="main_result.png" style="width:50%;"> Breakdown of database difference types by version (more verbose summaries of differences in the dataset, see below). | db_diff | code-generation (%)| original (%)| |:----------------------------------------------------------|------------------:|-----------:| | Database incorrectly updated. | 22.8 | 22.8 | | Database not updated at all but updates were required. | 33.6 | 14.4 | | Database updated when none was required for the task. | 5.6 | 7.6 | | No database: session terminated due to max steps reached. | 2 | 0.4 | | Perfect match with no required updates. | 33.2 | 32.4 | | Perfect match with required updates. | 2.8 | 22.4 | | Total | 100 | 100 | A more granular analysis by database table. | % | ('flights', 'code-generation') | ('flights', 'original') | ('reservations', 'code-generation') | ('reservations', 'original') | ('users', 'code-generation') | ('users', 'original') | |:-------------------------------------------------|---------------------------------:|--------------------------:|--------------------------------------:|-------------------------------:|-------------------------------:|------------------------:| | Not updated at all and no updates were required. | 74 | 74.7 | 10 | 10 | 34.7 | 34.7 | | Not updated at all but updates were required. | 21.3 | 6 | 46 | 16 | 43.3 | 19.3 | | Update required and correctly made. | 0 | 13.3 | 6 | 36.7 | 2.7 | 30.7 | | Update required and incorrectly made. | 1.3 | 4 | 36.7 | 36.7 | 16.7 | 13.3 | | Updated but no updates were required. | 2 | 1.3 | 0 | 0 | 1.3 | 1.3 | | No database produced. | 1.3 | 0.7 | 1.3 | 0.7 | 1.3 | 0.7 | | Total | 100 | 100 | 100 | 100 | 100 | 100 | Preliminary analysis suggests that code agents are capable of making the required updates but often fail to do so. For example, there were no cases where updates to available flight seating were correctly made, with attempts that were often slightly off. In the original version with tools, the logic is hard coded into the method booking flights. ## Dataset Rationale The purpose of this dataset is to allow further investigation into the differences, advantages and disadvantages of using code agents versus agents with action spaces more contrained by bespoke tools. You can find another example [here](https://huggingface.co/datasets/snorkelai/Multi-Turn-Insurance-Underwriting-Code-Gen). ## Dataset Structure The dataset contains one rollout for each of the 50 tasks in the original environment, for each version (original and code-agent versions). Rollouts were produced for the following models: - GPT-5 (medium) - Claude Sonnet 4.5 - Gemini 2.5 Pro (default settings) - Kimi K2 Instruct - Qwen3-Coder 480B A35B Instruct All dataset fields are munged information from the original repo, with a few additional highlighted: - `task_id (string)`: Unique identifier of task from original Tau Bench 2 repo. - `model (string)`: The AI assistant model used to generate responses. - `version (string)`: Either `original` or `code-generation` for the code-agent version. - `user_scenario (dict)`: Task-level information from original repo. - `db_update_required (bool)`: Whether or not the task requires an update to the database (determined by differencing expected hashes). - `trace (list of dictionaries in json format)`: Contains the full trace including an initial question, many steps of reasoning interleaved with questions/answers between the assistant and underwriter, and a final answer to the question. - Each step in the trace contains (all values serialized): - `turn_idx`: Turn number for that trace - `role`: The role for that step - user or assistant - `content`: The text content of that step - 'time_stamp': Timestamp - 'tool_calls': Optional data about tool use - `reward (float)`: Overall reward computed with default evaluators (see below for how we did this for the code agents). - `reward_breakdown (dict)`: Breakdown of reward. - `db_diff (string)`: Brief summary of differences between expected and actual database after interaction. - `db_diff_by_table (dict)`: Summary of differences between expected and actual database for each table. - `db_diff_verbose (string)`: More verbose summary of the differences in the expected and actual databases when updates were required but incorrect. ## Code Agent Adaptation We made the code-agent version by removing all tools from the environment except one mocking an action represented by pure text. We replaced the rest with a code interpreter, which we confirmed could be used to successfully update the database as required. ### Code interpreter tool ``` from langchain_experimental.tools import PythonAstREPLTool ... def code_evaluation(self, expression: str) -> str: """ Execute a Python expression, assuming access to variables: 'db': the Pydantic model representing the database. the database is a Pydantic model with information about passengers, flights, reservations etc. 'data_models': a dictionary containing relevant Pydantic models useful for updating the database, 'airport_codes': a variable representing airport codes. Each entry in 'airport_codes' is a dictionary mapping IATA codes to AirportInfo objects. HINT: To see the output of an expression you need to print it. Output is required to be fewer than 100,000 tokens, otherwise an error will be returned. Args: expression (str): Python expression with 'db' and 'airport_codes' variables Returns: str: result of the expression """ custom_locals = {'db': self.db, 'data_models': self.data_models, 'airport_codes': self.airport_codes} repl_tool = PythonAstREPLTool(locals=custom_locals) try: result = repl_tool.run(expression) token_count = token_counter(text=result) if token_count > 100_000: result = f"TOOL ERROR: {token_count} tokens in output of code execution, output needs to be < 100,000 tokens. Please execute code that produces results under that limit." else: pass except Exception as e: result = f"TOOL ERROR: {str(e)[:1000]}..." return result ``` ## Code Agent Evaluation The original Tau Bench 2 repo computes reward online. For database checks (is the database updated appropriately if at all), this means executing expected tool use patterns to derive the expected updated database. We therefore refactored evaluation to occur offline with expected database hashes generated from the original version with tool use. We confirmed in a handful of examples that with a code interpreter, models are capable of making the same updates. We also evaluated the differences in generated and expected database updates using GPT-5 when updates were made at all, provided in the `db_diff_summary (string)` field. #### Personal and Sensitive Information The dataset was developed based on synthetic information so there is no sensitive data risk. ## Citation If you find this dataset helpful, please cite us: ``` @misc{snorkelai2025tau2benchcodeagents, author = {Snorkel AI}, title = {Tau^2 Bench Airline With Code Agents}, year = {2025}, howpublished = {\url{https://huggingface.co/datasets/snorkelai/Tau2-Bench-Airline-With-Code-Agents}}, } ```
A dataset of multi-turn interaction sample traces and associated metadata based on the Airline environment from Tau^2 Bench, including both the original version and the version with code agents, providing task ID, model type, version, user scenario, and other information.
