CanvasA11y: Bridging the Accessibility Semantic Voids in Mobile Apps via Multimodal LLMs

CanvasA11y: Automated Accessibility Remediation for Canvas-Based GUIs Overview This artifact accompanies the paper CanvasA11y, a framework that automatically reconstructs accessibility semantics for visually rendered, canvas-based graphical user interfaces (GUIs) in Android applications. Modern mobile applications frequently render UI elements directly onto a canvas, bypassing the native accessibility tree. As a result, assistive technologies (e.g., screen readers) cannot interpret or interact with these interfaces. CanvasA11y addresses this limitation by transforming raw pixels into structured, interactive accessibility nodes and automatically generating executable remediation code. Key Features End-to-End Pipeline: Detects inaccessible regions, extracts semantic elements, aligns spatial information, and synthesizes executable accessibility code. MLLM-Driven Semantic Extraction: Uses a Multimodal Large Language Model (MLLM) to infer human-centric UI semantics from visual input. Deterministic Code Generation: Produces reproducible Android accessibility patches using structured intermediate representations. Anti-Occlusion Mechanism: Ensures precise touch handling in overlapping UI elements via area-based prioritization. Pipeline Overview CanvasA11y consists of four sequential phases: Phase I – Semantic Void DetectionIdentifies regions in the UI that lack accessibility metadata. Phase II – Visual-Semantic ExtractionConverts cropped GUI images into structured semantic nodes using an MLLM. Phase III – Spatial AlignmentMaps normalized coordinates to absolute screen space. Phase IV – Automated Code SynthesisGenerates executable Android code using ExploreByTouchHelper. Repository Structure . ├── phase1_detection/ # Semantic void detection ├── phase2_mllm_extraction/ # MLLM-based GUI parsing ├── phase3_alignment/ # Coordinate transformation ├── phase4_codegen/ # Accessibility code synthesis │ └── codegen.py ├── examples/ # Sample inputs and outputs └── README.md Input Format (Phase III Output) The code generator expects a JSON file containing aligned virtual nodes: [ { "virtual_tree": [ { "id": 1, "label": "May 25, Available", "abs_bounds": [100, 200, 400, 500] } ] } ] Usage Step 1: Prepare Input Place the Phase III output in: phase3_virtual_tree.json Step 2: Run Code Generation python codegen.py Step 3: Output The system generates: CanvasA11yHelper.java This file can be directly integrated into an Android project. Generated Code Characteristics The synthesized Java class: Extends ExploreByTouchHelper Implements: getVirtualViewAt(float x, float y) onPopulateNodeForVirtualView(int id, AccessibilityNodeInfoCompat node) Injects semantic labels as accessibility descriptions Implements automatic hit-testing logic Anti-Occlusion Strategy To resolve interaction ambiguity in overlapping UI elements, CanvasA11y applies an Area-Based Anti-Occlusion mechanism: Each virtual node's bounding box area is computed Nodes are sorted in ascending order of area Touch events are resolved from smallest to largest element This ensures that deeply nested interactive elements are prioritized during hit-testing. Example Given two overlapping elements: A large calendar cell A small event indicator inside it The generated code guarantees that touches on the overlapping region correctly select the smaller, semantically precise element. Reproducibility The deterministic code generation pipeline does not rely on runtime randomness MLLM-based components (Phase II) are configured with temperature = 0 All intermediate representations are explicitly serialized Dependencies Python 3.8+ Android SDK (for integration) Optional: OpenAI API (for MLLM-based extraction) Limitations Assumes accurate detection and cropping in earlier phases Complex animations or dynamic canvases are not fully supported Semantic inference quality depends on the MLLM Citation If you use this artifact, please cite: @inproceedings{canvasa11y2026, title={CanvasA11y: Automated Accessibility Remediation for Canvas-Based GUIs}, author={Anonymous}, booktitle={xxxxxx}, year={2026} } License This artifact is released under the MIT License. Contact For questions or issues, please contact the authors via the corresponding paper submission.