LLMShot: Reducing snapshot testing maintenance via LLMs

LLMShot A tool for analyzing UI snapshot differences using LLMs. This project helps identify visual differences between expected and actual UI screenshots, categorize the differences, simulate ignoring them, and generate comprehensive analysis reports. Accepted Paper https://arxiv.org/abs/2507.10062: @misc{kaynak2025llmshotreducingsnapshottesting, title={LLMShot: Reducing snapshot testing maintenance via LLMs}, author={Ergün Batuhan Kaynak and Mayasah Lami and Sahand Moslemi and Anil Koyuncu}, year={2025}, eprint={2507.10062}, archivePrefix={arXiv}, primaryClass={cs.SE}, url={https://arxiv.org/abs/2507.10062}, } Requirements Python 3.8+ Pillow NumPy colorama Ollama (with gemma3:4b and gemma3:12b models installed) Xcode 16.3 Requirements Installation Install required Python packages: pip install numpy pillow colorama Install Ollama: Follow the instructions at ollama.ai to install Ollama Pull the required models: For "gemma3:4b": ollama pull gemma3:4b For "gemma3:12b": ollama pull gemma3:12b General Content The xcode_project/ directory contains all Xcode-related files including the iOS app and tests for dataset creation. There are also several scripts provided to recreate our experiments: generate_dataset.sh: Converts Xcode-generated snapshots to a dataset with the expected format and outputs to the dataset/ directory. process_snapshots.py: Main experiment script that processes data from the dataset/ directory. Runs LLMs (which may take a long time) and outputs results to the analysis_results/ directory. generate_report.py: Creates an HTML report for analysis results, see Reported Features section. Output results to reports/ directory. Detailed instructions are provided below for dataset creation, running experiments, and generating reports. Snapshotting and Dataset Creation A readily available dataset is given in the dataset/ directory. You can create your own dataset via: Create a dataset directory in the project root Execute tests in Xcode Run ./generate_dataset.sh Each test directory should contain: reference.png: Expected UI state failure.png: Current UI state with potential issues diff.png: Visualization of differences metadata.json: Test details and categories Xcode project was tested with Xcode 16.3. Extending The Dataset Our testing framework utilizes XCTest and XCUITest, Apple's native UI testing tools, for simulating UI actions. We also extend the base XCUIElement and XCUIApplication behaviour to allow for commonly used snpshot test actions. These include navigation actions (like vertical scrolling), handling time‑dependent events (such as waiting for loading or obscuring elements, or navigation transitions), and managing launch arguments. Each test is configured to run the app in the default configuration, then relaunch the app with set of feature flags. Here, both setupAction() and snapshotAction() functions are the same for both test runs, and they are overridden by each individual test to define their own setup and sequence of UI actions. To extend the dataset, one can Attach labels to the views using .accessibilityIdentifier(), which can then be interacted with forceTapElement(\_:). Create new test classses and cases that override setupAction() and snapshotAction() functions to define their own set of UI actions. Define their own set of test flags in FeatureFlags.swift and subsequently write the logic change in the main program. These flags are then passed during test cases for use during automated snapshot generation. Run generate_dataset.sh to convert created snapshots to the expected structure for analysis. This script looks for snapshots inside the snapshot save directory (by default /SnapshotInstructorUITests/__Snapshots__/BaseUITest) to find pairs of reference and failure snapshots, then creates their diff. Please see the existing test code and dataset for examples. Experiment Execution We provide a CLI interface for recreating our experiments. To run the tool with no arguments and use the interactive menu: python process_snapshots.py This will present a menu with these options: Select Model: Choose between 4b or 12b models Run Standard Analysis: Analyze all snapshots and identify differences Run 'Ignore Reason' Analysis (From Analysis): Ignore the main difference found in the standard analysis Run 'Ignore Reason' Analysis (From Metadata): Ignore the first category from metadata Run 'Analyze and Ignore' Analysis: Detect and ignore the main difference in a single step To run all analyses with both models in batch mode: python process_snapshots.py --all This will: Run all four analysis modes with the 4b model Run all four analysis modes with the 12b model Generate a comprehensive report Report Generation After running analyses, generate a visual HTML report via: python generate_report.py The report will be created in the reports/snapshot_report_<timestamp>.html file and automatically opened in your default browser. The reported features are as follows: Metrics Dashboard: See aggregated accuracy and performance metrics across all analysis modes Test Case Browser: View individual test cases with images and detailed analysis Visual Comparison: Compare reference and failure images with highlighted differences