Dolci-Instruct-SFT-Tool-Use-Fixed

# Dolci-Instruct-SFT-Tool-Use-Fixed ## Dataset Description **Dolci-Instruct-SFT-Tool-Use-Fixed** is a cleaned and re-formatted version of the [`allenai/Dolci-Instruct-SFT-Tool-Use`](https://huggingface.co/datasets/allenai/Dolci-Instruct-SFT-Tool-Use) tool-use dataset. It is designed as the **tool-calling (function-calling) extension** of the [`openbmb/UltraData-SFT-2605`](https://huggingface.co/datasets/openbmb/UltraData-SFT-2605) Supervised Fine-Tuning dataset, so that tool-use samples can be mixed into UltraData-SFT-2605 training runs seamlessly. The raw Dolci-Instruct-SFT-Tool-Use data uses a custom message schema (`role=environment`, `function_calls` / `functions` as raw strings). This repository **cleans** that schema into the standard **OpenAI `messages` format**, and **re-aligns the table-level fields** (`id`/`dataset_source`) to match UltraData-SFT-2605's field names (`uid`/`source`, plus `domain` and `think_type`). The data is also **converted from Parquet to JSON** (JSON Lines) for easy `datasets`-loader usage. ## Relationship to the source datasets | Dataset | Role | Link | | --- | --- | --- | | `allenai/Dolci-Instruct-SFT-Tool-Use` | Upstream tool-use data (this repo's raw source) | [🤗 HF](https://huggingface.co/datasets/allenai/Dolci-Instruct-SFT-Tool-Use) | | `allenai/Dolci-Instruct-SFT` | Main Dolci-Instruct dataset card | [🤗 HF](https://huggingface.co/datasets/allenai/Dolci-Instruct-SFT) | | `openbmb/UltraData-SFT-2605` | Target format / the SFT dataset this extends | [🤗 HF](https://huggingface.co/datasets/openbmb/UltraData-SFT-2605) | This dataset is a **drop-in tool-call supplement** for UltraData-SFT-2605: its `domain` field is set to `tool_call` and its `think_type` is `no_think`, so it can be concatenated with UltraData-SFT-2605 configs without any further preprocessing. > **Note:** These additions are intended to be used together with, not as replacements for, existing [`openbmb/UltraData-SFT-2605`](https://huggingface.co/datasets/openbmb/UltraData-SFT-2605) Supervised Fine-Tuning dataset (see the [UltraData technical report](https://arxiv.org/pdf/2602.09003)). It serves as an expanded, modern **tool-calling** foundation for training leading LLMs. ## What was changed (cleaning & conversion) The transformation is fully reproducible via `scripts/convertds.py`. For each sample: 1. **Field rename / alignment to UltraData-SFT-2605** - `id` → `uid` - `dataset_source` → `source` - added `domain = "tool_call"` and `think_type = "no_think"` (matching UltraData-SFT-2605's `no_think` split convention). 2. **Message normalization to standard OpenAI format** - `system`: the `functions` definitions are merged into the `content` text (appended as `"\n\nAvailable functions:\n{functions}"`), and the `functions` field is dropped. - `assistant`: the raw `function_calls` string is parsed into a proper `tool_calls` list, each with `id`, `type: "function"`, and `function: {name, arguments}`. Empty assistant messages (no content and no calls) are removed. - `environment`: converted to `role = "tool"`, each response assigned a matching `tool_call_id`. Parallel tool responses that were concatenated with `\n` are split back into individual `tool` messages. - all redundant fields (`functions`, `function_calls`) are removed. 3. **Format validation** - Each normalized sample is validated to ensure it only contains valid roles (`system`/`user`/`assistant`/`tool`), that every `assistant` message has content or tool calls, and that every `tool` message carries a `tool_call_id`. 4. **Format conversion** - Parquet (`data/train-*.parquet`) → JSON Lines (`data/train-*.json`, `orient="records", lines=True`). ### Example (normalized sample) ```json { "uid": "olmo-toolu-sft-mix-T2-S2-f2-bfclv3-decontaminated_S2_179340", "messages": [ {"role": "system", "content": "You are a helpful function-calling AI assistant. ...\n\nAvailable functions:\n[{\"type\": \"function\", ...}]"}, {"role": "user", "content": "Can you compare the 5-day weather forecasts for Paris and Madrid?"}, {"role": "assistant", "content": null, "tool_calls": [ {"id": "call_7218ca5c", "type": "function", "function": {"name": "weather.forecast_weather_api", "arguments": "{\"q\": \"Paris\", \"days\": 5}"}}, {"id": "call_31be866c", "type": "function", "function": {"name": "weather.forecast_weather_api", "arguments": "{\"q\": \"Madrid\", \"days\": 5}"}} ]}, {"role": "tool", "tool_call_id": "call_7218ca5c", "content": "{\"results\": {\"forecast\": [...]}}"}, {"role": "tool", "tool_call_id": "call_31be866c", "content": "{\"results\": {\"forecast\": [...]}}"}, {"role": "assistant", "content": "Overall, Madrid will be warmer than Paris over the next 5 days."} ], "source": "allenai/olmo-toolu-sft-mix-T2-S2-f2-bfclv3-decontaminated", "domain": "tool_call", "think_type": "no_think" } ``` ## Schema Each row has the following table-level fields (aligned with UltraData-SFT-2605): | Field | Type | Description | | --- | --- | --- | | `uid` | `string` | Unique sample identifier. | | `messages` | `list` | The conversation in OpenAI `messages` format. | | `source` | `string` | Original source dataset of the sample. | | `domain` | `string` | Capability domain, fixed to `tool_call`. | | `think_type` | `string` | Thinking mode, fixed to `no_think`. | ### `messages` element structure Each element of `messages` is a dictionary containing: | Field | Type | Description | | --- | --- | --- | | `role` | `string` | One of `system`, `user`, `assistant`, `tool`. | | `content` | `string` (nullable) | Message text; `null` for assistant messages that only issue tool calls. | | `tool_calls` | `list` (optional) | Present on `assistant` messages that invoke functions. | | `tool_call_id` | `string` (optional) | Present on `tool` messages, linking to a `tool_calls[].id`. | Each item in `tool_calls` is: ```json { "id": "call_xxxxxxxx", "type": "function", "function": { "name": "weather.forecast_weather_api", "arguments": "{\"q\": \"Paris\", \"days\": 5}" } } ``` ## Dataset statistics - **Total examples**: 227,579 - **Split**: `train` only - **Format**: JSON Lines (`data/train-*.json`), 6 shards - **Size on disk**: ~2.92 GB (raw JSON) ## How to load from Hugging Face ```python from datasets import load_dataset ds = load_dataset("nekocyrene/Dolci-Instruct-SFT-Tool-Use-Fixed", split="train") # Combine with UltraData-SFT-2605 (same field schema) ultradata = load_dataset("openbmb/UltraData-SFT-2605", "Code", split="no_think") mixed = concatenate_datasets([ultradata, ds]) ``` ### Reproducing the conversion ```bash # Convert one parquet shard to JSON python scripts/convertds.py \ data/train-00000-of-00006.parquet \ -o data/train-00000-of-00006.json \ --input-format parquet \ --output-format json ``` ## License This project is released under the [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) license. Note that the upstream `allenai/Dolci-Instruct-SFT-Tool-Use` dataset is licensed under **ODC-BY**, and is intended for research and educational use in accordance with Ai2's [Responsible Use Guidelines](https://allenai.org/responsible-use). Users are responsible for complying with the license terms of all upstream datasets. ## Citation If you use this dataset, please cite the original work: ```bibtex @misc{olmo2025olmo3, title={Olmo 3}, author={Team Olmo and Allyson Ettinger and Amanda Bertsch and Bailey Kuehl and David Graham and David Heineman and Dirk Groeneveld and Faeze Brahman and Finbarr Timbers and Hamish Ivison and Jacob Morrison and Jake Poznanski and Kyle Lo and Luca Soldaini and Matt Jordan and Mayee Chen and Michael Noukhovitch and Nathan Lambert and Pete Walsh and Pradeep Dasigi and Robert Berry and Saumya Malik and Saurabh Shah and Scott Geng and Shane Arora and Shashank Gupta and Taira Anderson and Teng Xiao and Tyler Murray and Tyler Romero and Victoria Graf and Akari Asai and Akshita Bhagia and Alexander Wettig and Alisa Liu and Aman Rangapur and Chloe Anastasiades and Costa Huang and Dustin Schwenk and Harsh Trivedi and Ian Magnusson and Jaron Lochner and Jiacheng Liu and Lester James V. Miranda and Maarten Sap and Malia Morgan and Michael Schmitz and Michal Guerquin and Michael Wilson and Regan Huff and Ronan Le Bras and Rui Xin and Rulin Shao and Sam Skjonsberg and Shannon Zejiang Shen and Shuyue Stella Li and Tucker Wilde and Valentina Pyatkin and Will Merrill and Yapei Chang and Yuling Gu and Zhiyuan Zeng and Ashish Sabharwal and Luke Zettlemoyer and Pang Wei Koh and Ali Farhadi and Noah A. Smith and Hannaneh Hajishirzi}, year={2025}, eprint={2512.13961}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2512.13961}, } @misc{ultradata-sft-2605, title={UltraData-SFT-2605}, author={OpenBMB}, year={2026}, url={https://huggingface.co/datasets/openbmb/UltraData-SFT-2605}, publisher={Hugging Face} } ```