Rager: A Rocket-based time series feature transformer for fast skeleton-based action and gesture recognition

Description This is a set of 21 skeletal-based action and gesture recognition datasets. The datasets have been processed into the standardized NPZ format for the experiments experiments carried out as part of the paper "Rager: A Rocket-based time series feature transformer for fast skeleton-based action and gesture recognition". Python script for loading and processing the dataset: import numpy as np dataset = np.load('KARD.npz', allow_pickle=True) data = dataset['data'] labels = dataset['labels'] data # array of subsets/subjects data[0] # array of gesture repetitions from the first subset/subject data[0][0] # 2D array MxN, where M is the number of features (fixed) and N is the number of frames (variable) # Features are arranged as follows: [x1, y1, z1, x2, y2, z2, ..., xM, yM, zM], where x, y, z are body landmark coordinates data[1][2][3,4] # subject 1, gesture repetition 2, feature 3, frame 4 (float) labels[0] # class number (str) '1', '2' etc. Source papers of the datasets and sites from which their original versions can be downloaded ETRI_S1_C1 paper: Jang, J., Kim, D., Park, C., Jang, M., Lee, J., Kim, J.: Etri-activity3d: A large-scale rgb-d dataset for robots to recognize daily activities of the elderly. In: 2020IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS),pp. 10990–10997 (2020). https://doi.org/10.1109/IROS45743.2020.9341160 site: https://github.com/ai4r/AIR-Action-Recognition FLORENCE paper: Seidenari, L., Varano, V., Berretti, S., Del Bimbo, A., Pala, P.: Recognizingactions from depth cameras as weakly aligned multi-part bag-of-poses. In: 2013IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp.479–485 (2013). https://doi.org/10.1109/CVPRW.2013.77 site: https://www.micc.unifi.it/resources/datasets/florence-3d-actions-dataset/ FPHAB paper: Garcia-Hernando, G., Yuan, S., Baek, S., Kim, T.-K.: First-person hand actionbenchmark with rgb-d videos and 3d hand pose annotations. In: 2018 IEEE/CVFConference on Computer Vision and Pattern Recognition, pp. 409–419 (2018).https://doi.org/10.1109/CVPR.2018.00050 site: https://guiggh.github.io/publications/first-person-hands/ G3D_enhanced paper: Bloom, V., Makris, D., Argyriou, V.: G3d: A gaming action dataset and realtime action recognition evaluation framework. In: 2012 IEEE Computer SocietyConference on Computer Vision and Pattern Recognition Workshops, pp. 7–12(2012). https://doi.org/10.1109/CVPRW.2012.6239175 site: https://www-shadoc.irisa.fr/oad-datasets/ KARD paper: Gaglio, S., Re, G.L., Morana, M.: Human activity recognition process using 3-d posture data. IEEE Transactions on Human-Machine Systems 45(5), 586–597(2015) https://doi.org/10.1109/THMS.2014.2377111 site: https://data.mendeley.com/datasets/k28dtm7tr6/1 LIBRAS paper: Dua, D., Graff, C.: UCI Machine Learning Repository.http://archive.ics.uci.edu/ml (2017, (accessed on 16 May 2026)) site: https://archive.ics.uci.edu/dataset/181/libras+movement LSA64 paper: Ronchetti, F., Quiroga, F., Estrebou, C., Lanzarini, L., Rosete, A.: Lsa64: Adataset of argentinian sign language. XX II Congreso Argentino de Ciencias dela Computaci´on (CACIC) (2016) site: https://facundoq.github.io/datasets/lsa64/ MAD paper: Huang, D., Yao, S., Wang, Y., De La Torre, F.: Sequential max-margin eventdetectors. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ComputerVision – ECCV 2014, pp. 410–424. Springer, Cham (2014) site: https://www-shadoc.irisa.fr/oad-datasets/ MSRA paper: Li, W., Zhang, Z., Liu, Z.: Action recognition based on a bag of 3d points. In: 2010IEEE Computer Society Conference on Computer Vision and Pattern Recognition- Workshops, pp. 9–14 (2010). https://doi.org/10.1109/CVPRW.2010.5543273 site: https://www.kaggle.com/datasets/ghazouanihaythem/msraction3dskeleton20joints MSRC12_M1, MSRC12_M2, MSRC12_M3, MSRC12_M4, MSRC12_M5, MSRC6 paper: Fothergill, S., Mentis, H., Kohli, P., Nowozin, S.: Instructing people for train-ing gestural interactive systems. In: Proceedings of the SIGCHI Conference onHuman Factors in Computing Systems. CHI ’12, pp. 1737–1746. Association forComputing Machinery, New York, NY, USA (2012). https://doi.org/10.1145/2207676.2208303 . https://doi.org/10.1145/2207676.2208303 site: https://www-shadoc.irisa.fr/oad-datasets/ NTU_60_S1_C1 paper: Shahroudy, A., Liu, J., Ng, T.-T., Wang, G.: NTU RGB+D: A Large ScaleDataset for 3D Human Activity Analysis . In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1010–1019. IEEE Computer Society,Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.115 site: OAD paper: Li, Y., Lan, C., Xing, J., Zeng, W., Yuan, C., Liu, J.: Online human actiondetection using joint classification-regression recurrent neural networks. In: Leibe,B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision – ECCV 2016, pp.203–220. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46478-7 13 site: https://www-shadoc.irisa.fr/oad-datasets/ SYSU paper: Hu, J.-F., Zheng, W.-S., Lai, J., Zhang, J.: Jointly learning heterogeneous fea-tures for RGB-D activity recognition. IEEE Transactions on Pattern Analysis andMachine Intelligence 39(11), 2186–2200 (2017) https://doi.org/10.1109/TPAMI.2016.2640292 site: https://web.archive.org/web/20250420183914/https://isee-ai.cn/~hujianfang/ProjectJOULE.html UTD paper: Chen, C., Jafari, R., Kehtarnavaz, N.: Utd-mhad: A multimodal dataset forhuman action recognition utilizing a depth camera and a wearable inertial sensor.In: 2015 IEEE International Conference on Image Processing (ICIP), pp. 168–172(2015). https://doi.org/10.1109/ICIP.2015.7350781 site: https://www.kaggle.com/datasets/dasmehdixtr/human-action-recognition-dataset UTK paper: Xia, L., Chen, C.-C., Aggarwal, J.K.: View invariant human action recognitionusing histograms of 3d joints. In: 2012 IEEE Computer Society Conference onComputer Vision and Pattern Recognition Workshops, pp. 20–27 (2012). https://doi.org/10.1109/CVPRW.2012.6239233 site: https://cvrc.ece.utexas.edu/KinectDatasets/HOJ3D.html VISAPP paper: Celebi, S., Aydin, A.S., Temiz, T.T., Arici, T.: Gesture recognition using skeletondata with weighted dynamic time warping. VISAPP 2013 - Proceedings of theInternational Conference on Computer Vision Theory and Applications 1, 620–625 (2013) https://doi.org/10.5220/0004217606200625 site: https://github.com/maroszII/TATS/blob/main/datasets/VISAPP%20dataset.zip