DPJAIT DATASET – Multimodal Dataset for Indoor 3D Drone Tracking

======================= License ======================= The DPJAIT dataset is made available under the CC BY 4.0 license https://creativecommons.org/licenses/by/4.0/ ======================= Summary ======================= DPJAIT DATASET – MULTIMODAL DATASET FOR INDOOR 3D DRONE TRACKING The DPJAIT dataset has been designed for research on vision-based 3D drone tracking. The dataset consists of real measurements registered by a Vicon system containing a synchronized RGB multicamera set and motion capture acquisition, as well as simulated sequences obtained from a similar but virtual camera system created in Unreal Engine and AirSim simulator. The scene for the simulation sequences was prepared using a model of the Human Motion Lab (HML) at the Polish-Japanese Academy of Information Technology (PJAIT) in Bytom, Poland, in which real sequences were registered. The DPJAIT dataset is available only for scientific purposes. It is obligatory to cite the following paper in every work that uses the dataset: J. Rosner, T. Krzeszowski, A. Świtoński, H. Josiński, W. Lindenheim-Locher, M. Zielinski, G. Paleta, M. Paszkuta, K. Wojciechowski: Multimodal dataset for indoor 3D drone tracking challenge, Scientific Data, 2024 ======================= Data description ======================= The dataset consists of 13 simulated and 18 real sequences, which differ in the number of drones and their pattern of moving on scene. The sequences were prepared in such a way that they could be used for various types of research. Some sequences contain a larger amount of drones but with limited motion or a smaller amount with a bigger degree of freedom. Additionally, some simulated sequences were generated based on measurements performed in a real laboratory, so they can be used to compare the results obtained for simulation and real sequences. The simulated sequences were created using an environment based on the Unreal Engine and the AirSim plugin. It is an open-source project created by Microsoft to provide high-fidelity simulation of a variety of autonomous vehicles. Inside the environment, a scene based on the laboratory where real-life recordings took place was created. At the simulation scene, eight different cameras were placed. For some sequences, the stage size was enlarged twice the size of the HML laboratory to accommodate more flying drones without an issue of potential collisions between each of them. This allowed the generation of sequences with a large number of drones (up to 10), which was not possible to achieve in real conditions. Five different drone models were used in the simulations. Most sequences contain data from eight cameras, except three sequences generated based on real sequences (S11_D4, S12_D3, S13_D3), which contain only data from four cameras. In addition, sequences S01_D2_A, S02_D4_A, and S03_D10_A contain images from the drone camera (First Person View, FPV), and ArUco markers placed on walls. In real data scenarios, drones are manually controlled by skilled operators and tracked by a multi-modal acquisition system. Videos are registered by a set of four RGB cameras -- cam_1, cam_2, cam_3, and cam_4 -- with 1924x1082 resolution, located in the corners of the lab. Moreover, motion capture measurements are used to provide reference locations and orientations. It is achieved by tracking four markers -- A, B, C, and D -- attached to the top of the drones and forming an asymmetrical cross (see files MarkersCross_1.jpg, MarkersCross_2.jpg, and MarkersCrosses.pdf in "Additional_Files" folder). Details on how to establish the location and orientation in case of the known 3D coordinates of the markers are described by Lindenheim-Locher, W. et al. (Lindenheim-Locher, W. et al. YOLOv5 Drone Detection Using Multimodal Data Registered by the Vicon System. Sensors 2023, 23, 6396. https://doi.org/10.3390/s23146396). Moreover, to distinguish different drones visible at the same time instant, various lengths of the cross arms are applied (see MarkersCrosses.pdf in the "Additional_Files" folder). Ground truth data were acquired using a Vicon motion capture system. Synchronization and calibration of the motion capture system and video cameras were carried out using software and hardware provided by Vicon. ======================= Dataset structure ======================= * Additional_Files - directory with additional files * lab_hml_map.pdf - scene diagram with camera placement * MarkersCross_1.jpg - placement of markers on the drone * MarkersCross_2.jpg - placement of markers on the drone * MarkersCrosses.pdf - diagrams with dimensions of crosses with markers * dl_data-ReadMe.txt - description of files with drones detections using the YOLOv5 model * Real_Data_ArUco - additional files for sequences with ArUco markers * ArUco-ReadMe.txt - file structure description * images with the arrangement of markers on the walls * Real_Data - 18 video sequences recorded in HML at the PJAIT. * 4 recordings from cameras placed on the scene * stationary_camera_data.csv - cameras calibration data for OpenCV camera model * .xcp - cameras calibration from Vicon motion capture system * .c3d - 3D coordinates of markers on crosses mounted on drones recorded by the Vicon system (see files MarkersCross_1.jpg, MarkersCross_2.jpg, and MarkersCrosses.pdf) * dl_data - drones detections using the YOLOv5 model * sequences with ArUco markers (_A in the name) additionally: * FPV recordings from drones camera * fpv_camera_data.csv - FPV camera parameters * ArUco_3D.xlsx - data of ArUco markers placed on the scene * _REF_ORI.csv - the drone's reference orientation corresponding to the data from the drone's camera * _REF_POS.csv - the drone's reference position corresponding to the data from the drone's camera * Simulated_Data - 13 simulation video sequences. * 4 to 8 recordings from cameras placed on the scene * stationary_camera_data.csv - cameras calibration data for OpenCV camera model * _pos_25.csv - position and orientation of the drone * _cam_25.csv (only sequences with ArUco markers - _A in the name) - position, orientation, and parameters of the drone's camera * drone_masks.zip - extracted drone masks * dl_data - drones detections using the YOLOv5 model * sequences with ArUco markers (_A in the name) additionally: * FPV recordings from the drone's camera * markersAruco.csv - data of ArUco markers placed on the scene ======================= Project participants ======================= Jakub Rosner Tomasz Krzeszowski <[e-mail]> (Rzeszow University of Technology) Adam Świtoński (Silesian University of Technology) Henryk Josiński (Silesian University of Technology) Wojciech Lindenheim-Locher Michał Zieliński Grzegorz Paleta Marcin Paszkuta (Silesian University of Technology) Konrad Wojciechowski (Polish-Japanese Academy of Information Technology) ======================= Acknowledgments ======================= This work has been supported by the National Centre for Research and Development within the research project "Innovative technology for creating multimedia events based on drone combat with synergy between the VR, AR and physical levels" in the years 2020–2023, Project No. POIR.01.02.00-00-0160/20. ======================= Further information ======================= For any questions, comments or other issues please contact Tomasz Krzeszowski <[e-mail]>. The dataset is protected by password until acceptance of the article in the Scientific Data Journal. After acceptance of the article the dataset will be available for all.