Replication Package for "Automated Program Repair for UI-centric Android Bugs: How Far are We?"

DroidFixBench — Artifact "Automated Program Repair of UI-centric Android Bugs: How Far are We?" This is the research artifact for the ISSTA 2026 paper above. It contains the DroidFixBench benchmark (real and synthetic UI-centric Android bugs), the source for the five Automated Program Repair (APR) tools we evaluate (ChatRepair, OpenHands, D4C, NTR, RewardRepair), the complete set of generated patches, the manual-analysis codebooks/taxonomy, and a Docker-based harness that runs each tool over the benchmark. Badges requested: Available, Functional, Reusable (see STATUS). Environment: see REQUIREMENTS. License:MIT (see LICENSE). Scope note. The artifact covers patch generation and inspection of results. Patch validation (compiling each patch and running the Android UI tests on an emulator) is documented but not containerized, because it needs a full Android SDK + emulator that does not run under Docker on commodity reviewer hardware. Part 1 — Getting Started Guide 1.1 What this artifact is A reproducibility package with two usable layers: The paper's results, ready to inspect. GeneratedPatches/ holds every patch each tool produced for both datasets; Codebooks/ holds the manual analysis protocols and the bug taxonomy. No setup is needed to read these. A runnable harness. docker/ builds one container per tool. A single adapter feeds the Dataset/ CSVs to each tool with a uniform CLI, so you can regenerate patches for a single bug or the whole benchmark. 1.2 Requirements Docker with the Compose plugin. The smoke test and the three API-based tools need only a CPU. NTR and RewardRepair need an NVIDIA GPU host. Full details: REQUIREMENTS. 1.3 Installation # 1. Clone (or unzip the archived artifact) and enter it git clone https://github.com/SageSELab/UI-Program-Repair.git cd UI-Program-Repair # 2. Start Docker (macOS: `open -a Docker`), then build the shared base # image and one lightweight tool image used for the smoke test. docker compose -f docker/docker-compose.yml build base docker compose -f docker/docker-compose.yml --profile api build chatrepair Building base + chatrepair downloads ~0.7 GB and takes a few minutes on a typical connection. No API keys are required for installation or the smoke test. 1.4 Smoke test (no API key, < 5 min) This confirms the harness is installed and the dataset wiring works by converting one benchmark bug into a tool's input — without calling any paid API or GPU. docker run --rm -v "$PWD/Dataset:/data:ro" \ --entrypoint python droidfixbench/chatrepair:latest \ /workspace/dataset_adapter.py \ --dataset real --bug realbug-8-setRunningActionMode --out-dir /tmp/smoke Expected output: [adapter] real: wrote 1 row(s) (bug 'realbug-8-setRunningActionMode') [adapter] code -> /tmp/smoke/code.csv [adapter] artifact -> /tmp/smoke/artifact.csv Seeing those three lines means the image built correctly, the Dataset/ CSVs are readable, and the adapter produced the per-tool input files. The entire Getting Started flow (install + smoke test) completes well within 30 minutes. (Optional, requires a paid key) To confirm a full generation call, add your OpenAI key and run one bug end-to-end: docker run --rm -e OPENAI_API_KEY=sk-... -v "$PWD/Dataset:/data:ro" \ droidfixbench/chatrepair:latest \ --dataset real --bug realbug-8-setRunningActionMode -- --total_tries 1 --chain_length 1 It prints the constructed repair prompt and the model's response. Part 2 — Step-by-Step Instructions 2.1 Paper claims supported by this artifact TODO (authors): fill in the exact table/figure/section numbers and the reported numbers from the camera-ready paper. The rows below map each claim to the artifact evidence; replace the bracketed TODO cells. # Paper claim Location in paper Supported by How to verify C1 Per-tool repair effectiveness on real bugs (plausible/correct patch counts) Table 4 GeneratedPatches/<Tool>/<Tool>-result-real-bugs.csv Inspect the CSVs; optionally re-generate with the Docker tools (§2.3) C2 Per-tool repair effectiveness on synthetic bugs Table 3 GeneratedPatches/<Tool>/<Tool>-result-synthetic-bugs.csv Same as C1 C3 Taxonomy of UI-centric bug / failure categories Figure 5 Codebooks/taxonomy-categories.csv Read the taxonomy CSV and the codebooks C4 Qualitative analysis of unsuccessful patches Section 5.3 Codebooks/Codebook_for_qualitative_analysis_of_the_unsuccessful_generated_patches.pdf+ the patch CSVs Follow the codebook against the generated patches C5 Correctness-annotation protocol for plausible patches Section 4.3 Codebooks/Codebook_for_annotating_correct_patches.pdf Read the codebook C6 DroidFixBench composition (50 real / 46 synthetic bugs, apps covered) Section 4.1 / Table 1 Dataset/*.csv Inspect dataset entries (50 real, 46 synthetic) 2.2 Paper claims NOT (fully) supported, and why End-to-end re-derivation of the headline numbers. The three LLM-based tools (ChatRepair, OpenHands, D4C) call non-deterministic, paid APIs, so a fresh run will not reproduce identical patches/counts. We therefore ship the exact generated patches used in the paper (GeneratedPatches/) for verification, and provide the harness to re-rungeneration if desired. Patch validation (pass/fail on the Android UI tests). Validation requires building each app and running instrumented UI tests on an Android emulator (ReplicationPackages/<Tool>/validator/junit.py,ReproducingScripts/app/). This is not containerized and needs an Android SDK + emulator; it is documented in §2.4 but excluded from the Docker harness. NTR / RewardRepair generation without a GPU. These need large model weights and an NVIDIA GPU (§2.3). Their images build and their data wiring is exercisable on CPU, but inference requires GPU hardware. RewardRepair input masking. The upstream data_converter.py/mask scheme was not part of the source we received; the adapter reconstructs a best-effort single-chunk masked input (clearly marked UNVERIFIED in dataset_adapter.py). 2.3 Reproducing / re-running patch generation Inspect the shipped results (no setup). The patches behind the paper are inGeneratedPatches/<Tool>/<Tool>-result-{real,synthetic}-bugs.csv. Re-generate with the harness. First configure secrets: cp docker/.env.example docker/.env # then edit docker/.env docker compose -f docker/docker-compose.yml build base docker compose -f docker/docker-compose.yml --profile api build Run any tool over a single bug or the whole dataset (uniform CLI): # One bug docker compose -f docker/docker-compose.yml run --rm \ chatrepair --dataset real --bug realbug-8-setRunningActionMode # Whole synthetic dataset (real API spend — see warning) docker compose -f docker/docker-compose.yml run --rm \ d4c --dataset synthetic --all -- --max_try 3 Output is written under out/<tool>/. The GPU tools (on an NVIDIA host, with model weights mounted and paths set in .env): docker compose -f docker/docker-compose.yml --profile gpu build docker compose -f docker/docker-compose.yml run --rm ntr --dataset real --all ⚠️ --all runs 50 (real bugs, broken down into 97 individual slugs to run the individual buggy methods for each bug) or 46 (synthetic) bugs × multiple model tries — this is real OpenAI/all-hands spend for the API tools. Start with --bug. Full harness documentation, the data-wiring details, and per-tool model instructions are in docker/README.md. 2.4 Patch validation (documented, not containerized) To validate generated patches by running the Android UI tests: ReproducingScripts/app/README.md describes building the app + instrumentation APKs and running them on an emulator (adb shell am instrument ...); each tool's validator/junit.py automates compile + adb install + test for a bug. This requires Android Studio / SDK (API 30, build-tools 30.0.2, AGP 4.1.1) and a running AVD emulator. 2.5 Running your own experiments (reuse) The harness is not limited to reproducing our runs: - Pick any bug or the full set: --dataset real|synthetic with--bug <slug> or --all. - Tune a tool: pass tool-native flags after -- (e.g. -- --max_try 3,-- --total_tries 5 --chain_length 1). - Swap models (NTR/RewardRepair): set NTR_MODEL_PATH / NTR_TARGET_MODULES / RR_MODEL_PATH. - Add a tool or dataset: the conversion lives in one place,docker/dataset_adapter.py (emit_* functions); add a Dockerfile + entrypoint following the existing pattern and a service in docker/docker-compose.yml. 2.6 Data provenance, ethics, and storage Provenance. Real bugs are drawn from open-source Android apps (e.g. GPSLogger, Harmonic Hacker News, SkyTube, PSLab, UrlChecker, TrickyTripper) and their public issue trackers; synthetic bugs are mutants seeded into those apps. Each record carries the buggy code, the developer fix, and the UI test. Ethics/legal. All subject apps are open source; the dataset and codebooks are released for academic use (MIT, see LICENSE), and upstream tool code retains its original license. No personal data is included. Storage. The repository is small (CSVs + PDFs). The .apk files are hosted externally (see "Apk Files" below). Container images total ~13 GB; GPU model weights (downloaded separately) can be tens of GB. Artifact layout Dataset/ — the DroidFixBench benchmark (real + synthetic) real-bug-buggy-fixed-code.csv, real-bug-test-cases.csv synthetic-bug-buggy-fixed-code.csv, synthetic-bug-test-cases.csv Apk Files — all .apk files for real and synthetic bugs: [link — TODO] ReplicationPackages/ — source for each APR tool (ChatRepair, D4C, NTR, OpenHands, RewardRepair); per-tool instructions inside each folder. docker/ — the containerized generation harness (Dockerfiles, the dataset adapter, compose, and docker/README.md). ReproducingScripts/ — Android test cases that reproduce the bugs and check related features; instructions in ReproducingScripts/app/. ShortScripts/ — scripts for processing and aggregating results. GeneratedPatches/ — all generated patches, per tool, for real and synthetic bugs (<Tool>/<Tool>-result-{real,synthetic}-bugs.csv). Codebooks/ — manual-analysis codebooks and the bug/error taxonomy (taxonomy-categories.csv).