Benchmarking data on worker reactions to triggering events

1. Real-world Benchmarking Data

The objective of this task was to determine if Virtual Reality-based captured behavioral data on responses to notifications are similar to what is expected in real-world settings. For this purpose, a real-world bench mark experiment was designed to capture participant response times to wearable watch alarms triggered upon simulated traffic near the mobile work zone on the experiment site in an urban setting. The proposed scope of data collection of the real-world study included the external environmental factors (e.g., site accessibility, weather). The key parameters of research are defined as reaction time to received alarms and the heart rate measures. Table 1 provides the list of parameters that were controlled and measured during the experiments.

Table 1. Key parameters measured and tracked during real-world experiments

Variable name
Descriptions

Key parameters captured
Reaction time
The time that one takes from getting the haptic or sound alarm from a wearable alarm device, herein referring to the apple watch, to the point when the participant gives a response by stopping the alarm by pressing on the screen of the smartwatch

Inter-beat interval (IBI, heart rate)
The time interval between individual beats of the heart; the data is measured by using E4 application provided by Empatica

External factors tracked
Ambient noise
The level of ambient noise in the area is a factor potentially influencing participants’ reactions and is considered in the experiment design

Temperature
Daytime temperature recorded at each experiment

Number of pedestrians on site
Number of participants counted during the time of the experiment to record on the varying factors in the external environment in real-world settings

In the experiment, each participant was asked to participate in the experiment three times. In each trial, data was recorded separately for each alarm sent to smartwatch from the administrator at triggering events (precisely, every time the remote-controlled toy car reaches the line 30 ft apart from the designated work area). Each alarm signal at each trial was recorded for all 31 participants to the experiment. Timestamps are automatically recorded in server in the events recorded in the format of Table 2: 

Table 2. Format of raw data stored in the server, starting in December 2022.

Timestamp
From
Event

0
2022-12-08 13:37:53.101391      
VR
Received car approaching alert, mode=3, id=1000

1
2022-12-08 15:53:05.098288
Watch
Start Simulation

2
2022-12-08 15:53:07.437488   
VR
Received car approaching alert, mode=4, id=1004

3
2022-12-08 15:53:13.064067
Watch
Stop Simulation

4
2022-12-08 15:53:13.163635
Watch
Stop Simulation

...

2417
2023-03-03 16:17:46.166644
Watch
1398

2418
2023-03-03 16:18:00.004425
Watch
1398

2419
2023-03-03 16:18.01.272071
Watch
1398

2420
2023-03-03 16:18:07.359187
Watch
Stop Simulation

2421
2023-03-03 16:18:07.388183 
Watch
Stop Simulation

Some intervals used different timestamps as benchmarks to calibrate on the vehicle speed and user response time to the alarm signals, which include the following cases: 

1) At the beginning of each trial, vehicle travels 70 ft from start point to the 30 ft apart point, when the first alarm is signaled; given this travel distance, the travel time of the first trip the toy vehicle makes is calculated by subtracting tn_alarm1_sent from tn_start.

2) Similarly, user response times to all alarms are recorded by subtracting the timestamps when the alarm is received by participant from when the alarm is sent from the server. (tn_alarmn_sent - tn_alarmn_received)

2. Supplementary Data

Ambient noise level data were collected using a noise meter, allowing to save noise level by seconds to multiple seconds (i.e., 5, 10, 30, 60 seconds). All noise data recorded were recorded in the interval of one second using the meter.
The collected data was processed to match the certain timestamps collected for user response time data collected in the experiment to allow comparisons and correlation analysis to be performed later on, which include the following: 1) worker response; 2) sending of alarm signals; 3) start and stop of experiments.
All data points were later modified using the rolling mean function of pandas python module to replace the missing data points by moving average method.