LLMShot: Reducing snapshot testing maintenance via LLMs
收藏资源简介：
LLMShot A tool for analyzing UI snapshot differences using LLMs. This project helps identify visual differences between expected and actual UI screenshots, categorize the differences, simulate ignoring them, and generate comprehensive analysis reports. Accepted Paper https://arxiv.org/abs/2507.10062: @misc{kaynak2025llmshotreducingsnapshottesting, title={LLMShot: Reducing snapshot testing maintenance via LLMs}, author={Ergün Batuhan Kaynak and Mayasah Lami and Sahand Moslemi and Anil Koyuncu}, year={2025}, eprint={2507.10062}, archivePrefix={arXiv}, primaryClass={cs.SE}, url={https://arxiv.org/abs/2507.10062}, } Requirements Python 3.8+ Ollama (with gemma3:4b and gemma3:12b models installed) Pillow NumPy colorama Xcode 16.3 Installation Clone the repository: git clone https://github.com/SahandMoslemi/LLMShot.git Install required Python packages: pip install numpy pillow colorama Install Ollama: Follow the instructions at ollama.ai to install Ollama Pull the required models: ollama pull gemma3:4bollama pull gemma3:12b Usage The xcode_project/ directory contains all Xcode-related files including the iOS app and tests for dataset creation. There are also several scripts provided to recreate our experiments: generate_dataset.sh: Converts Xcode generated snapshots to a dataset with the expected format, outputs to dataset/ directory. process_snapshots.py: Main experiment script. Outputs results to analysis_results/ directory. generate_report.py: Creates an HTML report for analysis results, see Reported Features section. Output results to reports/ directory. Snapshotting and Dataset Creation A readily available dataset is given in the dataset/ directory. You can create your own dataset via: Create a dataset directory in the project root Execute tests in Xcode Run generate_dataset.sh Each test directory should contain: reference.png: Expected UI state failure.png: Current UI state with potential issues diff.png: Visualization of differences metadata.json: Test details and categories Xcode project was tested with Xcode 16.3. Extending The Dataset Our testing framework utilizes XCTest and XCUITest, Apple's native UI testing tools, for simulating UI actions. We also extend the base XCUIElement and XCUIApplication behaviour to allow for commonly used snpshot test actions. These include navigation actions (like vertical scrolling), handling time‑dependent events (such as waiting for loading or obscuring elements, or navigation transitions), and managing launch arguments. Each test is configured to run the app in the default configuration, then relaunch the app with set of feature flags. Here, both setupAction() and snapshotAction() functions are the same for both test runs, and they are overriden by each individual test to define their own setup and sequence of UI actions. To extend the dataset, one can Attach labels to the views using .accessibilityIdentifier(), which can then be interracted with forceTapElement(\_:). Create new test classses and cases that override setupAction() and snapshotAction() functions to define their own set of UI actions. Define their own set of test flags in FeatureFlags.swift and subsequently write the logic change in the main program. These flags are then passed during test cases for use during auotomated snapshot generation. Run generate_dataset.sh to convert created snapshots to the expected structure for analysis. This script looks for snapshots inside the snapshot save directory (by default /SnapshotInstructorUITests/__Snapshots__/BaseUITest) to find pairs of reference and failure snapshots, then creates their diff. Please see the existing test code and dataset for examples. Running Experiments We provide a CLI interface for recreating our experiments. After running analyses, generate a visual HTML report via: python generate_report.py The report will be created in the reports/snapshot_report_<timestamp>.html file and automatically opened in your default browser. To run the tool with no arguments and use the interactive menu: python process_snapshots.py This will present a menu with these options: Select Model: Choose between 4b or 12b models Run Standard Analysis: Analyze all snapshots and identify differences Run 'Ignore Reason' Analysis (From Analysis): Ignore the main difference found in the standard analysis Run 'Ignore Reason' Analysis (From Metadata): Ignore the first category from metadata Run 'Analyze and Ignore' Analysis: Detect and ignore the main difference in a single step To run all analyses with both models in batch mode: python process_snapshots.py --all This will: Run all four analysis modes with the 4b model Run all four analysis modes with the 12b model Generate a comprehensive report Reported Features Metrics Dashboard: See aggregated accuracy and performance metrics across all analysis modes Test Case Browser: View individual test cases with images and detailed analysis Visual Comparison: Compare reference and failure images with highlighted differences
LLMShot是一款利用大语言模型（LLM）分析UI快照差异的工具。本项目可用于识别预期UI截图与实际UI截图之间的视觉差异，对差异进行分类，支持模拟忽略差异，并生成全面的分析报告。 ## 已接收论文 https://arxiv.org/abs/2507.10062： bibtex @misc{kaynak2025llmshotreducingsnapshottesting, title={LLMShot: Reducing snapshot testing maintenance via LLMs}, author={Ergün Batuhan Kaynak and Mayasah Lami and Sahand Moslemi and Anil Koyuncu}, year={2025}, eprint={2507.10062}, archivePrefix={arXiv}, primaryClass={cs.SE}, url={https://arxiv.org/abs/2507.10062}, } ## 系统要求 - Python 3.8及以上版本 - Ollama（需安装gemma3:4b与gemma3:12b模型） - Pillow - NumPy - colorama - Xcode 16.3 ## 安装步骤 1. 克隆仓库： git clone https://github.com/SahandMoslemi/LLMShot.git 2. 安装所需Python依赖包： pip install numpy pillow colorama 3. 安装Ollama：请遵循ollama.ai官方提供的指引完成Ollama安装 4. 拉取所需模型： ollama pull gemma3:4b ollama pull gemma3:12b ## 使用方法 xcode_project/ 目录包含所有与Xcode相关的文件，包括iOS应用及用于数据集创建的测试脚本。此外还提供了多个可用于复现本研究实验的脚本： - `generate_dataset.sh`：将Xcode生成的快照转换为符合规范格式的数据集，输出结果保存至dataset/目录 - `process_snapshots.py`：核心实验脚本，分析结果将输出至analysis_results/目录 - `generate_report.py`：为分析结果生成HTML报告，详见「报告功能」章节，生成的报告将保存至reports/目录 ## 快照生成与数据集创建 dataset/ 目录中已提供现成可用的数据集。你也可以通过以下步骤创建自定义数据集： 1. 在项目根目录创建dataset文件夹 2. 在Xcode中执行测试 3. 运行generate_dataset.sh脚本 每个测试目录需包含以下文件： - `reference.png`：预期UI状态快照 - `failure.png`：存在潜在问题的当前UI状态快照 - `diff.png`：差异可视化结果 - `metadata.json`：测试详情与分类信息 本Xcode项目已通过Xcode 16.3版本测试。 ## 扩展数据集 本测试框架依托Apple原生UI测试工具XCTest与XCUITest实现UI动作模拟，同时对XCUIElement与XCUIApplication的基础行为进行扩展，以支持常用的快照测试动作，包括导航动作（如垂直滚动）、时间相关事件处理（如等待加载、遮挡元素或导航过渡）以及启动参数管理。 每个测试均配置为以默认配置启动应用，随后通过一组功能标志重新启动应用。其中`setupAction()`与`snapshotAction()`函数在两次测试运行中保持一致，各独立测试可重写这两个函数以定义自定义的初始化流程与UI动作序列。 若要扩展数据集，可通过以下方式实现： 1. 使用`.accessibilityIdentifier()`为视图添加标签，随后可通过`forceTapElement(_:)`方法与之交互 2. 创建新的测试类与测试用例，重写`setupAction()`与`snapshotAction()`函数以定义自定义UI动作集 3. 在`FeatureFlags.swift`中定义自定义测试标志，并在主程序中编写对应的逻辑变更。这些标志将在测试用例执行期间传入，用于自动化快照生成 4. 运行`generate_dataset.sh`脚本，将生成的快照转换为符合分析要求的标准结构。该脚本默认会在快照保存目录（/SnapshotInstructorUITests/__Snapshots__/BaseUITest）中查找参考快照与故障快照对，并生成二者的差异图。 请参考现有测试代码与数据集作为示例。 ## 运行实验 我们提供了用于复现本研究实验的命令行界面。完成分析后，可通过以下命令生成可视化HTML报告： python generate_report.py 生成的报告将保存至`reports/snapshot_report_<timestamp>.html`文件，并自动在默认浏览器中打开。 若不带参数运行工具，将进入交互式菜单模式： python process_snapshots.py 该菜单包含以下选项： - 选择模型：可选4b或12b版本模型 - 执行标准分析：分析所有快照并识别差异 - 执行「忽略原因」分析（基于分析结果）：忽略标准分析中识别出的主要差异 - 执行「忽略原因」分析（基于元数据）：忽略元数据中指定的首个分类差异 - 执行「分析并忽略」分析：一步完成差异检测与忽略操作 若需以批量模式运行两种模型的所有分析流程，可执行： python process_snapshots.py --all 该命令将完成以下操作： - 使用4b模型运行全部四种分析模式 - 使用12b模型运行全部四种分析模式 - 生成综合分析报告 ## 报告功能 - 指标仪表板：查看所有分析模式下的聚合准确率与性能指标 - 测试用例浏览器：查看包含截图与详细分析内容的单个测试用例 - 可视化对比：对比参考快照与故障快照，并高亮显示差异区域
