snorkelai/Multi-Turn-Insurance-Underwriting-Code-Gen

--- license: apache-2.0 dataset_info: features: - name: primary id dtype: int64 - name: company task id dtype: int64 - name: assistant model name dtype: string - name: task dtype: string - name: trace list: - name: additional_kwargs dtype: string - name: content dtype: string - name: id dtype: string - name: response_metadata dtype: string - name: role dtype: string - name: tool_calls dtype: string - name: type dtype: string - name: usage_metadata dtype: string - name: reference answer dtype: string - name: correct dtype: int64 - name: company name dtype: string - name: annual revenue dtype: int64 - name: number of employees dtype: int64 - name: total payroll dtype: int64 - name: number of vehicles dtype: int64 - name: building construction dtype: string - name: state dtype: string - name: company description dtype: string - name: lob dtype: string splits: - name: train num_bytes: 81414049 num_examples: 1800 download_size: 20056108 dataset_size: 81414049 configs: - config_name: default data_files: - split: train path: data/train-* --- # Dataset Card for Multi-Turn-Insurance-Underwriting-Code-Gen This dataset is a variant of the [Multi-Turn-Insurance-Underwriting](https://huggingface.co/datasets/snorkelai/Multi-Turn-Insurance-Underwriting) dataset, in which models do not get access to any tools except a code interpreter and a pointer to the relevant file system. This helps us analyze how well models explore their environments. ## Environment Creation <img src="dataset_codegen_diagram.png" style="width:50%;height:50%;"> This diagram shows the architecture of how we create the dataset, with assistant responses interleaved with questions, ending with a final answer to the task. - The input is one of 6 distinct types of tasks represented by a seed question. - The underwriter then asks a detailed question related to that task. - The assistant co-pilot then reasons about the question, asking follow up questions when necessary and calling the code interpreter to get extra information from the data environment. Importantly, there are no other tools involved. - After multiple steps of reasoning and interaction with a user, the co-pilot produces a final solution for the task, along with a rationale. Specifically, instead of a library of bespoke tools on an MCP server, we simply enable models with a Python interpreter from the experimental langchain library: ``` from langchain_experimental.tools import PythonREPLTool ... assistant_tools = [PythonREPLTool()] ``` In order to orient agents towards the correct resources, we include a path to the data directory in the system prompt (the prompt is identical to the original otherwise): ``` ... TIPS ON TOOLS AND REQUIRED DATA: You have been given a Python interpreter. ALL of the required data is in '~/sandbox/multi-turn-insurance-underwriting-benchmark-generation-code-gen/env_data' so you can use that data with Python to help the underwriter. ... ``` ## Analysis <img src="insurance_model_perf_codegen.png" style="width:50%;height:50%;"> Interestingly, models are capable of solving even some of the more difficult tasks with only a code interpreter, exploring the data environment and making inferences by integrating those results with user input. GPT-5 (medium) and Claude Sonnet 4.5 are actually more accurate in final task solutions by a small margin compared with the original benchmark based on tools. ### Example From Claude Sonnet 4.5 Below are from the first 5 code blocks produced on a task that was successfully completed. Importantly, most of these steps are interleaved with user interactions (here showing just the code for visualization). This type of exploratory behavior -- reading the file system and querying database metadata proactively -- is required because the model is not given any information about data or even the nature of the different files (beyond file names): <img src="code_example.png" style="width:50%;height:50%;"> ## Dataset Details - **Curated by:** Snorkel AI - **License:** Apache-2.0 License ## Quickstart To load the dataset, use the following code: ``` from datasets import load_dataset ds = load_dataset("snorkelai/Multi-Turn-Insurance-Underwriting-Code-Gen") ```license: apache-2.0 dataset_info: features: - name: primary id dtype: int64 - name: company task id dtype: int64 - name: assistant model name dtype: string - name: task dtype: string - name: trace list: - name: additional_kwargs dtype: string - name: content dtype: string - name: id dtype: string - name: response_metadata dtype: string - name: role dtype: string - name: tool_calls dtype: string - name: type dtype: string - name: usage_metadata dtype: string - name: reference answer dtype: string - name: correct dtype: int64 - name: company name dtype: string - name: annual revenue dtype: int64 - name: number of employees dtype: int64 - name: total payroll dtype: int64 - name: number of vehicles dtype: int64 - name: building construction dtype: string - name: state dtype: string - name: company description dtype: string - name: lob dtype: string splits: - name: train num_bytes: 81414049 num_examples: 1800 download_size: 20056108 dataset_size: 81414049 configs: - config_name: default data_files: - split: train path: data/train-* # 多轮保险承保代码生成数据集卡片 本数据集是[多轮保险承保](https://huggingface.co/datasets/snorkelai/Multi-Turn-Insurance-Underwriting)数据集的变体，在此数据集中，模型仅可使用代码解释器（code interpreter）与指向相关文件系统的指针，无法访问其他任何工具。这一设置便于我们分析模型对自身所处环境的探索能力。 ## 环境构建 <img src="dataset_codegen_diagram.png" style="width:50%;height:50%;"> 该图示展示了本数据集的构建架构，助手的回复与用户的提问交替出现，最终以任务的最终解答收尾。 - 输入为6种不同类型任务中的一种，由种子提问表征。 - 承保员随后会提出与该任务相关的详细问题。 - 助手副驾驶随后会针对该问题进行推理，必要时提出跟进问题，并调用代码解释器从数据环境中获取额外信息。值得注意的是，本流程未涉及其他任何工具。 - 经过多轮推理与用户交互后，助手将输出任务的最终解决方案及相应的推理依据。 具体而言，本数据集并未使用MCP服务器上的定制工具库，而是仅为模型启用了来自实验性LangChain库的Python解释器： from langchain_experimental.tools import PythonREPLTool ... assistant_tools = [PythonREPLTool()] 为了引导智能体定位到正确的资源，我们在系统提示词中加入了数据目录的路径（除此之外，提示词与原始版本完全一致）： ... TIPS ON TOOLS AND REQUIRED DATA: You have been given a Python interpreter. ALL of the required data is in '~/sandbox/multi-turn-insurance-underwriting-benchmark-generation-code-gen/env_data' so you can use that data with Python to help the underwriter. ... ## 分析 <img src="insurance_model_perf_codegen.png" style="width:50%;height:50%;"> 值得注意的是，仅依靠代码解释器，模型即可完成部分难度较高的任务：通过探索数据环境，并将所得结果与用户输入相结合进行推理。与基于传统工具的原始基准测试相比，GPT-5（中型）与Claude Sonnet 4.5在最终任务解决方案的准确率上反而有小幅领先。 ### Claude Sonnet 4.5示例 以下内容来自某一成功完成的任务中生成的前5个代码块。值得注意的是，大多数步骤均与用户交互交替进行（此处仅展示用于可视化的代码）。这种探索性行为——主动读取文件系统并查询数据库元数据——是必要的，因为模型未被提供任何关于数据的信息，甚至无法知晓不同文件的本质（仅能获取文件名）： <img src="code_example.png" style="width:50%;height:50%;"> ## 数据集详情 - **整理方：** Snorkel AI - **许可证：** Apache-2.0许可证 ## 快速入门 若要加载本数据集，请使用以下代码： from datasets import load_dataset ds = load_dataset("snorkelai/Multi-Turn-Insurance-Underwriting-Code-Gen")