Dataset for "Are We Done Yet?": A Vision-Based Judge for Autonomous Task Completion of Computer Use Agents

This dataset accompanies the paper “Are We Done Yet?: A Vision-Based Judge for Autonomous Task Completion of Computer Use Agents.” It contains human-labeled macOS GUI tasks, execution trajectories from three Computer Use Agents (CUAs), and 5 Vision–Language Models (VLM) judgments about whether each task was successfully completed. Each example links a high-level natural-language task description with final desktop screenshots, low-level actions, and binary success labels (“done” / “not done”, and "max steps exeeded, but not done"), enabling research on task completion evaluation, feedback, and self-correction in CUA agents. Potential Use Cases Training and evaluating vision-based judges of GUI task completion Studying failure modes and self-correction of computer-use agents Building reward models and offline RL pipelines based on evaluator outputs Benchmarking new CUAs, evaluators, or feedback mechanisms on realistic desktop workflows Processing Details Tasks were manually designed across 42 built-in macOS applications to cover diverse interaction patterns (menus, dialogs, text input, navigation, filters, settings, etc.). For each task, three CUAs were run in a controlled macOS environment (virtual machine); their full trajectories (screenshots, actions, and reasoning) were recorded. Human annotator reviewed the final state for each run and labeled whether the task goal was truly achieved according to the natural-language description. Multiple VLMs were prompted in a zero-shot setting with the task description + final screenshot to produce binary “done/not done” labels and short rationales.