DragSense: Towrads Detecting Drag-Related Accessibility Issues in Mobile Apps via LLMs

DragSense: Towrads Detecting Drag-Related Accessibility Issues in Mobile Apps via LLMs Overview The script analyzes pairs of screenshots and XML layout files to determine if a user must perform continuous drag operations. It classifies the interaction into specific categories and assesses whether alternative interaction methods exist for accessibility purposes. Key Features Multi-modal Analysis: Processes both image files (.png, .jpg) and Android layout XMLs. Structured Output: Uses JSON Schema to ensure the AI returns strictly formatted data. Batch Processing: Automatically pairs files with matching names and processes entire directories. Resilience: Includes incremental logging (.jsonl) so progress is saved even if the script is interrupted. Prerequisites Python: 3.9 or higher. Dependencies: pip install openai httpx python-dotenv Environment: A .env file in the root directory containing your API credentials: OPENAI_API_KEY=your_api_key_here Configuration Before running, update the following variables inside drag_detect.py: Variable Description MODEL_NAME The model ID (e.g., gpt-4o or the custom model specified in code). DATA_DIR Path to the folder containing your .png and .xml pairs. OUTPUT_DIR Path where results and logs will be saved. Output Structure The script generates three primary files in your OUTPUT_DIR: results.jsonl: A line-by-line log of every successful detection. results_all.json: A formatted summary of all successful runs, including token usage statistics. failures.json: A record of any files that failed to process and the corresponding error messages. Detection Logic (Prompt) The AI is tasked with three specific objectives: Identification: Does a drag operation exist? Classification: Does it fall under Adjustment, Playback, Map, or Sliding Verification? Accessibility: Is the operation mandatory? Are there alternatives or voice feedback? Notes File Pairing: Ensure your images and XML files share the same base name (e.g., home_screen.png and home_screen.xml).