DiffSpot

<h2 align="center">DiffSpot: Can VLMs Spot Fine-Grained Visual Differences in Web Interfaces?</h2> <p align="center"> <a href="https://huggingface.co/datasets/tencent/DiffSpot">🤗 Dataset</a> | <a href="https://github.com/Tencent/DiffSpot">🐙 GitHub</a> | <a href="https://arxiv.org/abs/2605.29615">📄 arXiv</a> </p> Vision-language models excel at high-level image–text alignment — but can they spot a *subtle* visual change? **DiffSpot** puts this to the test on rendered web interfaces, where a localized change is both a clean probe of fine-grained perception and a practical requirement for GUI agents and design tools. Each example is a pair of screenshots differing by a single mutated CSS property; the model must say what changed. The catch: it's hard. Evaluating 13 frontier VLMs zero-shot, **even the best identifies only 40.7% of true changes**, and **Hard-tier recall stays below 23% for every model**. Difficulty is strongly property-dependent — across CSS operators, neither pixel magnitude nor CLIP distance reliably predicts recall. ## 🔔 Overview - **Code-driven** — mutate one CSS property of a target element in self-contained HTML, re-render, and record the changed property / element / magnitude as ground truth - **Grounding gate** — keeps only pairs whose rendered pixel difference is confined to the target element, so the label is exactly what changed on screen - **4,400 pairs** — 3,900 has-diff balanced across **13 CSS-property operators × 3 difficulty tiers**, plus **500 no-diff controls** for hallucination measurement - **Open-ended spot-the-difference** task, scored by an LLM-as-Judge against the structured mutation log ## 👀 Example Pairs Each pair differs by a single CSS-level mutation. The model is given both screenshots and must name the change. <table> <tr><th width="50%">Before</th><th width="50%">After</th></tr> <tr> <td><img src="assets/examples/easy-fontsize-before.png"/></td> <td><img src="assets/examples/easy-fontsize-after.png"/></td> </tr> <tr><td colspan="2" align="center">_{The caption under the "PDF Viewer" heading is slightly larger.}</td></tr> <tr> <td><img src="assets/examples/easy-rounded-before.png"/></td> <td><img src="assets/examples/easy-rounded-after.png"/></td> </tr> <tr><td colspan="2" align="center">_{The "SOHBETE BAŞLA" button has more rounded corners.}</td></tr> <tr> <td><img src="assets/examples/easy-opacity-before.png"/></td> <td><img src="assets/examples/easy-opacity-after.png"/></td> </tr> <tr><td colspan="2" align="center">_{The card with the heading "Interesting Facts About Steamboat Springs" is much more transparent.}</td></tr> <tr> <td><img src="assets/examples/easy-gradient-before.png"/></td> <td><img src="assets/examples/easy-gradient-after.png"/></td> </tr> <tr><td colspan="2" align="center">_{The banner background showing the "Find your ideal partner" tagline now uses a different gradient.}</td></tr> <tr> <td><img src="assets/examples/easy-fontweight-before.png"/></td> <td><img src="assets/examples/easy-fontweight-after.png"/></td> </tr> <tr><td colspan="2" align="center">_{The subtitle "your loan installment!" in the financing section is slightly bolder.}</td></tr> </table> ## 📦 Loading ```python from datasets import load_dataset ds = load_dataset("tencent/DiffSpot", split="test") ex = ds[0] ex["image_before"] # PIL.Image ex["image_after"] # PIL.Image ex["ground_truth_diff"] # natural-language description of the change ``` ## 🗂️ Schema | Field | Type | Description | |---|---|---| | `id` | string | Unique example id | | `image_before` / `image_after` | image | Screenshot pair (PNG) | | `user_query` | string | The benchmark prompt | | `ground_truth_diff` | string | Natural-language description of the change | | `mutations_text` | list[string] | Templated mutation description (empty for no-diff) | | `mutation_types` | list[string] | Mutation operator name(s) | | `mutation_dicts_json` | list[string] | JSON-encoded structured mutation detail | | `task_type` | string | `visual_diff` or `no_diff` | | `difficulty` | string | `easy` / `medium` / `hard` / `no_diff` | | `domain` | string | One of 15 page-domain categories | | `pixel_diff` / `target_diff` / `outside_diff` | number | Pixel-change magnitudes | | `target_bbox_x/y/w/h` | int | Mutated element's bounding box (`-1` when not applicable) | Zero-shot evaluation of 13 frontier VLMs. **Easy / Med / Hard / Diff** are Recall on the 3,900 has-diff pairs (per tier and overall); **No-Diff** is specificity on the 500 control pairs; **Overall** is per-case accuracy `(TP + TN) / 4400` — the headline score. **Bold** = best overall; <u>underline</u> = best open-weight. | Model | Params | Easy | Med | Hard | Diff | No-Diff | Overall | |---|---|---:|---:|---:|---:|---:|---:| | *Open-weight* | | | | | | | | | Kimi K2.5 | 1T / 32B | 54.2 | 36.4 | 18.6 | <u>36.4</u> | 87.2 | <u>42.2</u> | | Qwen3.5-VL-397B | 397B / 17B | 45.1 | 31.5 | 13.7 | 30.1 | 96.6 | 37.6 | | Qwen3-VL-235B-Thinking | 235B / 22B | 30.1 | 17.3 | 10.5 | 19.3 | 98.8 | 28.3 | | GLM-4.6V-Flash | 9B | 24.5 | 17.6 | 9.3 | 17.1 | 75.8 | 23.8 | | GLM-4.6V | 106B / 12B | 17.0 | 10.9 | 5.5 | 11.2 | 99.6 | 21.2 | | Qwen3-VL-30B-Instruct | 30B / 3B | 14.5 | 9.0 | 4.5 | 9.3 | 82.0 | 17.6 | | Qwen3-VL-30B-Thinking | 30B / 3B | 16.5 | 8.8 | 3.8 | 9.7 | 77.8 | 17.5 | | Qwen3-VL-235B-Instruct | 235B / 22B | 9.6 | 3.0 | 2.6 | 5.1 | **100.0** | 15.9 | | InternVL3.5-30B-A3B | 30B / 3B | 4.7 | 3.9 | 3.8 | 4.2 | **100.0** | 15.0 | | *Proprietary* | | | | | | | | | Gemini 3.1 Pro | — | **60.5** | **38.9** | **22.7** | **40.7** | 98.4 | **47.2** | | Gemini 3 Flash | — | 52.5 | 32.5 | 18.2 | 34.4 | 91.4 | 40.9 | | Claude Opus 4.7 | — | 41.2 | 30.5 | 21.8 | 31.2 | 99.6 | 38.9 | | GPT-5.4 | — | 48.8 | 30.5 | 12.2 | 30.5 | 99.6 | 38.3 | Even the best model misses ~3 of every 5 ground-truth changes, and Hard-tier recall stays below 23% for every model; open-ended spot-the-difference on real web UIs is far from solved. ## 🔦 Citation ```bibtex @misc{zhang2026diffspotvlmsspotfinegrained, title={DiffSpot: Can VLMs Spot Fine-Grained Visual Differences in Web Interfaces?}, author={Linhao Zhang and Aiwei Liu and Yuan Liu and Xiao Zhou}, year={2026}, eprint={2605.29615}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2605.29615}, } ``` ## 📜 License Released under the MIT License (© 2026 Tencent). See [LICENSE](./LICENSE).