Cyclist Actions: Motion History Images and Trajectories

The dataset consists of over 1.1 million samples of labeled cyclists actions. Every sample consists of two motion history images (MHI), consisting of motion over the past second, from two different cameras, the past trajectory of the cyclist of the last second (50 past positions), and a label of the currently performed action.

The samples were extracted from 1,639 video sequences of cyclists moving across an urban intersection at the University of Applied sciences in Aschaffenburg: https://www.th-ab.de/ueber-uns/organisation/labor/kooperative-automatisierte-verkehrssysteme/ausstattung/

The uploaded files consist of a numpy file with MHI, trajectories and labels,  and a json file containing 5-fold cross validation/test split.

Every MHI has a size 192x192x2. The two channels contain the separate motion contours of the bike and the cyclist.

The manually created labels are:

straight: cyclists is moving and not turning

tr/tl: cyclist is turning left/right

move: cyclist is moving with nearly constant velocity and not turning

start: cyclist was standing and starts moving

starting_movement: first movement of cyclist before starting

stop: cyclist was moving/starting and slows down to a halt

wait: cyclist is standing

hand_signal_left/right: cyclist indicates a turn by hand signal

shoulder_check_left/right: cyclist looks over left/right shoulder

out_of_saddle: cyclist is standing

The numpy file consists of python dictionaries with scenes of the form:

{SCENE_NAME:
'mhi_hk1/2': [...], # list of zip compressed, python pickled mhi from cameras 1 and 2
'x/y/z_tracked': [...], # tracked cyclists positions in x/y/z directions,
'x/y/z_smoothed': [...], # smoothed (by rts smoother) cyclists positions in x/y/z directions,
'orientation': [...], # orientation of the cyclists estimated by kalman filters
'ts': [...], # utc timestamps in micro seconds
LABEL_NAME: [...], # labels of different actions (0 or 1)}

The zipped/pickled mhi can be decompressed using:

import numpy as np
import pickle
import zlib

d = np.load('mhi_dataset_0,1,2,3,4,6,9,13,18,26,33,41,49.npy', allow_pickle=True, encoding='latin1').item()
for scene_name in d:
for i in range(len(d[scene_name]['mhi_hk1']) - 50): # ommit 50 first frames as they have no mhi
mhi1 = pickle.loads(zlib.decompress(d[scene_name]['mhi_hk1'][49 + i].encode('latin1')), encoding='latin1')
mhi2 = pickle.loads(zlib.decompress(d[scene_name]['mhi_hk2'][49 + i].encode('latin1')), encoding='latin1')

This work results from the project DeCoInt 2, supported by the German Research Foundation (DFG) within the priority program SPP 1835: "Kooperativ interagierende Automobile", grant numbers DO 1186/1-2, FU 1005/1-2, and SI 674/11-2. Additionally, the work is supported by "Zentrum Digitalisierung Bayern".

Due to privacy laws in germany, we are not permitted to publish raw image data.