FR-GESTURE

The FR-GESTURE dataset consists of 3312 RGBD pairs corresponding to 12 hand signals for guiding UGVs in first response scenarios.

It was captured by two Intel RealSense depth cameras positioned in two different heights (low and high) with arbitrary perception angle. Each participant was asked to perform each gesture at 6 or 7 different distances, to increase diversity and therefore make the classifiers robust to multi-distance gesture recognition.

Two experimental protocols are introduced:

Uniform protocol (uniform_protocol_3312.json): The samples are randomly distributed into train/val/test partitions, with 2304/504/504 samples respectively. The classes within each split are equally distributed to avoid issues due to class imbalance.

Subject independent protocol (subject_independent_protocol_3312.json): This protocol aims to evaluate the generalization ability of the models to unseen subjects. Each split contains instances performed by different signers (who do not participate in more than one split). The train (s2, s3, s4, s5, s6), val (s1) and test (s0) splits contain 2304, 504 and 504 samples, respectively.

Information for each sample (including camera height and approximate distance) can be found in the entire dataset json (entire_dataset_3312.json).

The Classes folder contains the definition of each class. 

The dataset was collected in the context of TRIFFID project [1] and related study on vision-based gesture recognition [2]. 

Please consider citing the related paper [3] if you find this dataset useful for your purposes.

We would like to thank many members of HUA Computer Vision Group for participating in the data gathering procedure.

Aiming to respect participants' privacy rights, interested researchers should send an email at [e-mail] requesting the dataset and declaring that the data contained in FR-GESTURE will be leveraged only for research purposes, compatible with the original purpose of the collection, i.e., training rescue robots.

Feel free to contact us for any reason at [e-mail].

[1] Cani J, Koletsis P, Foteinos K, Kefaloukos I, Argyriou L, Falelakis M, Del Pino I, Santamaria-Navarro A, Čech M, Severa O, Umbrico A. TRIFFID: Autonomous Robotic Aid For Increasing First Responders Efficiency. In2025 6th International Conference in Electronic Engineering & Information Technology (EEITE) 2025 Jun 4 (pp. 1-9). IEEE.

[2] Foteinos K, Cani J, Linardakis M, Radoglou-Grammatikis P, Argyriou V, Sarigiannidis P, Varlamis I, Papadopoulos GT. Visual Hand Gesture Recognition with Deep Learning: A Comprehensive Review of Methods, Datasets, Challenges and Future Research Directions. arXiv preprint arXiv:2507.04465. 2025 Jul 6.

[3] Foteinos, K., Angelidis, G., Psiris, A., Argyriou, V., Sarigiannidis, P., and Papadopoulos, G. T., “FR-GESTURE: An RGBD Dataset For Gesture-based Human-Robot Interaction In First Responder Operations”, arXiv e-prints, arXiv preprint arXiv:2602.17573, 2026. doi:10.48550/arXiv.2602.17573.