LLMShot: Reducing snapshot testing maintenance via LLMs

LLMShot A tool for analyzing UI snapshot differences using LLMs. This project helps identify visual differences between expected and actual UI screenshots, categorize the differences, simulate ignoring them, and generate comprehensive analysis reports. Accepted Paper https://arxiv.org/abs/2507.10062: @misc{kaynak2025llmshotreducingsnapshottesting, title={LLMShot: Reducing snapshot testing maintenance via LLMs}, author={Ergün Batuhan Kaynak and Mayasah Lami and Sahand Moslemi and Anil Koyuncu}, year={2025}, eprint={2507.10062}, archivePrefix={arXiv}, primaryClass={cs.SE}, url={https://arxiv.org/abs/2507.10062}, } Requirements Python 3.8+ Ollama (with gemma3:4b and gemma3:12b models installed) Pillow NumPy colorama Xcode 16.3 Installation Install required Python packages: pip install numpy pillow colorama Install Ollama: Follow the instructions at ollama.ai to install Ollama Pull the required models: ollama pull gemma3:4bollama pull gemma3:12b Usage The xcode_project/ directory contains all Xcode-related files including the iOS app and tests for dataset creation. There are also several scripts provided to recreate our experiments: generate_dataset.sh: Converts Xcode generated snapshots to a dataset with the expected format, outputs to dataset/ directory. process_snapshots.py: Main experiment script. Outputs results to analysis_results/ directory. generate_report.py: Creates an HTML report for analysis results, see Reported Features section. Output results to reports/ directory. Snapshotting and Dataset Creation A readily available dataset is given in the dataset/ directory. You can create your own dataset via: Create a dataset directory in the project root Execute tests in Xcode Run ./generate_dataset.sh Each test directory should contain: reference.png: Expected UI state failure.png: Current UI state with potential issues diff.png: Visualization of differences metadata.json: Test details and categories Xcode project was tested with Xcode 16.3. Extending The Dataset Our testing framework utilizes XCTest and XCUITest, Apple's native UI testing tools, for simulating UI actions. We also extend the base XCUIElement and XCUIApplication behaviour to allow for commonly used snpshot test actions. These include navigation actions (like vertical scrolling), handling time‑dependent events (such as waiting for loading or obscuring elements, or navigation transitions), and managing launch arguments. Each test is configured to run the app in the default configuration, then relaunch the app with set of feature flags. Here, both setupAction() and snapshotAction() functions are the same for both test runs, and they are overriden by each individual test to define their own setup and sequence of UI actions. To extend the dataset, one can Attach labels to the views using .accessibilityIdentifier(), which can then be interracted with forceTapElement(\_:). Create new test classses and cases that override setupAction() and snapshotAction() functions to define their own set of UI actions. Define their own set of test flags in FeatureFlags.swift and subsequently write the logic change in the main program. These flags are then passed during test cases for use during auotomated snapshot generation. Run generate_dataset.sh to convert created snapshots to the expected structure for analysis. This script looks for snapshots inside the snapshot save directory (by default /SnapshotInstructorUITests/__Snapshots__/BaseUITest) to find pairs of reference and failure snapshots, then creates their diff. Please see the existing test code and dataset for examples. Running Experiments We provide a CLI interface for recreating our experiments. After running analyses, generate a visual HTML report via: python generate_report.py The report will be created in the reports/snapshot_report_<timestamp>.html file and automatically opened in your default browser. To run the tool with no arguments and use the interactive menu: python process_snapshots.py This will present a menu with these options: Select Model: Choose between 4b or 12b models Run Standard Analysis: Analyze all snapshots and identify differences Run 'Ignore Reason' Analysis (From Analysis): Ignore the main difference found in the standard analysis Run 'Ignore Reason' Analysis (From Metadata): Ignore the first category from metadata Run 'Analyze and Ignore' Analysis: Detect and ignore the main difference in a single step To run all analyses with both models in batch mode: python process_snapshots.py --all This will: Run all four analysis modes with the 4b model Run all four analysis modes with the 12b model Generate a comprehensive report Reported Features Metrics Dashboard: See aggregated accuracy and performance metrics across all analysis modes Test Case Browser: View individual test cases with images and detailed analysis Visual Comparison: Compare reference and failure images with highlighted differences