Cyclist Actions: Optical Flow Sequences and Trajectories

The dataset consists of over 1.1 million samples of labeled cyclists actions. Every sample consists of two optical flow sequences, recorded over the past second (9 optical flow images each), from two different cameras, the past trajectory of the cyclist of the last second (50 past positions), and a label of the currently performed action.

The samples were extracted from 1,639 video sequences of cyclists moving across an urban intersection at the University of Applied sciences in Aschaffenburg: https://www.th-ab.de/ueber-uns/organisation/labor/kooperative-automatisierte-verkehrssysteme/ausstattung/

The uploaded files consist of an archive containing 27 numpy files, a single numpy file containing trajectories only, and a json file containing 5-fold cross validation/test split.

The numpy files consist of python dictionaries with scenes of the form:

{SCENE_NAME:
'of_hk1/2': [...], # zip compressed, python pickled optical flow sequences of cameras 1/2
'x/y/z_tracked': [...], # tracked cyclists positions in x/y/z directions,
'x/y/z_smoothed': [...], # smoothed (by rts smoother) cyclists positions in x/y/z directions,
'orientation': [...], # orientation of the cyclists estimated by kalman filters
'ts': [...], # utc timestamps in micro seconds
LABEL_NAME: [...], # labels of different actions (0 or 1)}

The manually created labels are:

straight: cyclists is moving and not turning

tr/tl: cyclist is turning left/right

move: cyclist is moving with nearly constant velocity and not turning

start: cyclist was standing and starts moving

starting_movement: first movement of cyclist before starting

stop: cyclist was moving/starting and slows down to a halt

wait: cyclist is standing

hand_signal_left/right: cyclist indicates a turn by hand signal

shoulder_check_left/right: cyclist looks over left/right shoulder

out_of_saddle: cyclist is standing

The optical flow sequences were created using PWC-Net [1].

To extract the zipped/pickled optical flow sequences:

import cv2 as cv
import zlib
import pickle
import numpy as np

# visualize flow
def vis_of(of):
hsv = np.zeros([of.shape[0], of.shape[1], 3], dtype=np.uint8)
hsv[..., 1] = 255
mag, ang = cv.cartToPolar(of[..., 0].astype(np.float32), of[..., 1].astype(np.float32))
hsv[..., 0] = ang * 180 / np.pi / 2
hsv[..., 2] = cv.normalize(mag, None, 0, 255, cv.NORM_MINMAX)
bgr = cv.cvtColor(hsv, cv.COLOR_HSV2BGR)
return bgr

# load npy file from dataset
npy_path = 'of_dataset_0.npy'
data = np.load(npy_path, allow_pickle=True).item()
scene = data[list(data.keys())[0]]
# extract optical flow sequence
ofs = pickle.loads(zlib.decompress(scene['of_hk1'][i])).astype(np.float16) * 2.0 / 255.0 - 1.0

# show of images in sequence
for j in range(len(ofs)):
# create bgr image from 2 channel optical flow
bgr = vis_of(ofs[j])
cv.imshow("of", bgr)

Python code and a description to read the dataset can be found in our GitHub: https://github.com/CooperativeAutomatedTrafficSystemsLab/CyclistActionRecognition

[1] D. Sun, X. Yang, M. Liu, and J. Kautz, “PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, June 2018, pp. 8934–8943.

This work results from the project DeCoInt 2, supported by the German Research Foundation (DFG) within the priority program SPP 1835: "Kooperativ interagierende Automobile", grant numbers DO 1186/1-2, FU 1005/1-2, and SI 674/11-2. Additionally, the work is supported by "Zentrum Digitalisierung Bayern".

Due to privacy laws in germany, we are not permitted to publish image sequences.