Participant Study Dataset in Virtual Reality Integrated CARLA Simulator - 2. Test Series

Ensuring the safety of autonomous vehicles in dynamic environments hinges on accurately modeling interactions with vulnerable road users (VRUs). Traditional testing methods often fall short in replicating realistic human behavior, while collecting real-world data in safety-critical situations poses significant ethical and practical challenges.

To address this, we present a human-in-the-loop simulation framework that seamlessly integrates virtual reality (VR) with the CARLA driving simulator. This setup allows for the study of pedestrian-vehicle interactions within a controlled yet immersive environment. By leveraging VR-based motion tracking, our approach captures natural pedestrian movements and decision-making processes, thereby enhancing the realism of scenario-based testing.

Using the VR-integrated CARLA simulator, we conducted a user study with 24 participants (10 participants <30 years old, 14 participants >50 years old) equipped with Meta Quest Pro headsets. Participants were instructed to consecutively reach each point shown below within the CARLA Town01 environment, while multiple vehicles passed along the street. We designed four distinct scenario variants by introducing a distraction task for the pedestrian and varying the pedestrian’s initial spawning position:

Variation 1: Pedestrian spawned at the first position — pedestrian has a distraction task

Variation 2: Pedestrian spawned at the first position — pedestrian has not a distraction task

Variation 3: Pedestrian spawned at the second position — pedestrian has a distraction task

Variation 4: Pedestrian spawned at the second position — pedestrian has not a distraction task

Note: The variations are shuffled during the test runs. Please check the variations permutation list for the exact implementation order of the variations.

Each scenario could be repeated multiple times. Data collected for each run includes scene images from the ego vehicle’s perspective, alongside a CSV file containing:

Timestamp (datetime)

Pedestrian avatar’s position in CARLA Town01 (posx, posy, posz)

Vehicles' positions in CARLA Town01 (posx, posy, posz)

Participant’s head rotation (pitch, yaw, roll)

Metadata about the participants profile (sex, age, height, eye glasses)

Variations permutation list