VLM-based semantic monitoring for robotic agents

This dataset contains annotated execution traces for evaluating specification-grounded semantic monitoring of robotic manipulation tasks with vision-language models. The dataset includes simulated and physical executions of pick-and-place and object-sorting tasks. Each trace is associated with behavior-driven validation specifications, scenario parameters, execution videos, and ground-truth outcome labels. The data was used to evaluate whether VLM-based oracle assessments can determine if observed robotic executions satisfy or violate specification-derived task outcomes. The dataset contains three subsets: Pick-and-place in simulation: 560 executions generated from Gherkin-based scenario variants in NVIDIA Isaac Sim. Executions feature Emika Panda and UR10 robots and are recorded from three synchronized camera views. Object sorting in simulation: 280 long-horizon sorting executions involving multiple pick-and-place actions and a final sorting condition. Object sorting in the physical environment: 43 real-robot executions using a Kinova Gen3 setup with external video recordings and associated ground-truth annotations. The traces include execution videos, scenario parameters, task outcomes, and validation metadata used to compare VLM predictions against ground truth. The dataset supports research on runtime verification, robotic test oracles, vision-language model evaluation, and specification-grounded monitoring of robotic behavior. This dataset accompanies the paper: “Specification-Grounded Semantic Monitoring of Robotic Execution Traces with Vision-Language Models”