VC-Tooler-SFT

# VC-Tooler-SFT Supervised cold-start trajectories for **VC-Tooler: Learning Compositional and Adaptive Visual Tool Use**. ## 🔗 Links - 📄 **Paper**: [arXiv](#) <!-- TODO: add paper link --> - 🌐 **Project Page**: [w1zheng.github.io/VC-Tooler](https://w1zheng.github.io/VC-Tooler) - 🤗 **Hugging Face**: [VC-Tooler-SFT](https://huggingface.co/datasets/5551z/VC-Tooler-SFT) *(this dataset)* · [VC-Tooler-RL](https://huggingface.co/datasets/5551z/VC-Tooler-RL) - 🧩 **ModelScope**: [VC-Tooler-SFT](https://modelscope.cn/datasets/W1zheng/VC-Tooler-SFT) *(this dataset)* · [VC-Tooler-RL](https://modelscope.cn/datasets/W1zheng/VC-Tooler-RL) This dataset is the **Stage I (supervised fine-tuning)** trajectory bank used to teach a vision–language model to use visual tools as a *compositional* and *adaptive* capability, rather than to memorize a fixed set of invocation patterns. Each example is a multi-step, ReAct-style trajectory in which language reasoning is interleaved with tool calls and tool-returned observations while solving a visual query. <p align="center"> <img src="assets/data_composition.png" width="100%" alt="VC-Tooler SFT trajectory composition across Single-Tool, Multi-Tool, and Diverse-Tool-Context subsets."><br> <em>Composition of the SFT trajectory bank across the Single-Tool, Multi-Tool, and Diverse-Tool-Context capability levels.</em> </p> ## What's in this dataset VC-Tooler frames visual tool use around three core capabilities, and the trajectory bank is synthesized to provide supervision for each of them: - **Single-Tool (ST) grounding** — interpreting a tool schema, grounding a call in the visual input, and predicting valid arguments to execute one tool. - **Multi-Tool (MT) composition** — chaining several tool calls across steps, where each step reasons over the observations returned by earlier calls. - **Diverse-Tool-Context (DTC)** — the same underlying visual operation re-expressed through many different tool interfaces, so the model learns to adapt to novel schemas instead of memorizing tool identities. A small number of **no-tool reasoning** samples are also included to preserve the base model's direct-answering ability. ## How it was built <p align="center"> <img src="assets/synthesis_pipeline.png" width="100%" alt="Hierarchical trajectory synthesis pipeline: plan-then-execute, then diverse tool-context reinstantiation, producing Single-Tool, Multi-Tool, and Diverse-Tool-Context supervision."><br> <em>Hierarchical synthesis pipeline: a planner proposes a tool sequence, an executor grounds it into a tool-grounded trajectory (M=1 Single-Tool, M&gt;1 Multi-Tool), and reinstantiation re-expresses steps under diverse tool schemas.</em> </p> Trajectories are produced by a **hierarchical synthesis pipeline** with two complementary phases: 1. **Plan-then-Execute.** A large thinking VLM first proposes an ordered tool-use plan for a query (conditioned on the answer annotation, which is used only during planning to avoid answer leakage into execution). A non-thinking model then follows the plan step by step inside a tool environment, grounding each call and recording the returned observation. Trajectories are labeled ST or MT according to the number of tool calls. 2. **Diverse Tool-Context Reinstantiation.** For each executed step, a strong VLM inspects the pre-/post-action visual states and either matches the operation to an existing tool or defines a new schema, producing an alternative interface that realizes the *same* visual operation. This yields the DTC subset and greatly expands interface diversity. Candidate source instances are selected for **diversity**, **verifiability**, and **agentic relevance** (favoring queries that genuinely benefit from acquiring, transforming, or revisiting visual evidence) via a hierarchical filtering pipeline. Public multimodal sources include, among others, LLaVA-OneVision, DeepVision, and VisualProbe for plan-then-execute synthesis; Zebra-CoT and Monet for the diverse-context trajectories; and ChartVerse for the no-tool reasoning samples. After synthesis, trajectories are post-filtered to remove answer-inconsistent, format-invalid, or inefficient rollouts, and distractor tools are introduced into the candidate pool for robustness. The executable tools underlying the ST/MT subsets cover common visual operations (e.g., zoom, rotate, enhance, code execution, mark, and edit), while the DTC subset augments these with a large pool of mock-but-valid tool interfaces to drive adaptive tool-use learning. <p align="center"> <img src="assets/tool_wordcloud.png" width="30%" alt="Word cloud of tool names appearing across the trajectory bank."><br> <em>Tool-name diversity across the trajectory bank, spanning perception, editing, computation, and search operations.</em> </p> ## Intended use - Cold-start supervised fine-tuning of vision–language models for agentic, multi-step visual tool use. - Research on compositional and adaptive tool calling, schema generalization, and trajectory synthesis for multimodal agents. This bank is the supervised precursor to reinforcement learning; the companion RL dataset is released separately as **VC-Tooler-RL-Data**. ## Data format The released columns are: | Field | Type | Description | |---|---|---| | `id` | int64 | Trajectory identifier. | | `messages` | struct of two parallel lists: `role: list<string>`, `content: list<string>` | The ReAct-style turns. Turn *i* is `(role[i], content[i])`. | | `images` | list of struct `{bytes: binary, path: string}` | Query image(s) followed by any tool-produced visual states, embedded inline. `path` is the original relative path, kept for provenance only. | | `tools` | string | JSON-encoded list of tool schemas available for this trajectory (name, description, parameters). Parse with `json.loads`. | `messages.role` uses a tool-use protocol: | Role | Meaning | |---|---| | `system` | Task framing and tool-use protocol instructions. | | `user` | The visual query. | | `function_call` | Reasoning followed by a `<tool_call>` block containing the tool name and arguments. | | `observation` | The environment's response to the preceding call. | | `assistant` | Final answer, conditioned on the accumulated observations. | `function_call` / `observation` pairs repeat for multi-step trajectories. Images are referenced from the text by an `<image>` placeholder — in the `user` turn for query images and in `observation` turns for tool-produced ones — where the *n*-th placeholder corresponds to `images[n]`. ## License Released under **CC BY-NC 4.0** (non-commercial). Individual source datasets used during synthesis retain their own licenses and terms of use; please review and comply with the terms of each upstream source before use. ## Citation ```bibtex @inproceedings{vctooler2026, title = {VC-Tooler: Learning Compositional and Adaptive Visual Tool Use}, author = {Wu, Yizheng and Hua, Jiashen and Deng, Bing and Ye, Jieping}, booktitle = {arXiv}, year = {2026} } ```