Thermal Urban Feature Segmentation - Multispectral (RGB + Thermal) UAS-based images from Germany with annotations

The Thermal Urban Feature Segmentation (TUFSeg) dataset consists of annotated and combined RGB and thermal images, acquired by uncrewed aircraft system (UAS). They show various nighttime scenes from two German cities: Munich and Karlsruhe. Owing to the high overlap of 88%, only select images were annotated to prevent duplicate instances. See the "Usage" section below for details about the stored formats made available here.

The raw images were recorded with a 4k normal (RGB) and a FLIR-XT2 (thermal) camera using both DJI M600 Pro and M300 RTK uncrewed aircraft (UA). The RGBs have a resolution of 4000 x 3000 pixels, the thermals 640 x 512. All images were registered to a uniform format (3000 x 3750 pixels) to match thermal image aspect ratio and prevent RGB data loss. The Munich images were recorded during 8 p.m. to 6 a.m. in December 2019 with temperatures ranging from -5 °C to 2 °C, the Karlsruhe ones between 12 a.m. and 3 a.m. in January and March 2022 with temperatures between 0 °C and 3 °C. 

The dataset consists of 793 labelled images - 700 from Munich, 93 from Karlsruhe - from 14 UAS flights. A total of 8,010 common urban feature classes are annotated: buildings - 1,404, cars (cold) - 2,532, cars (warm) - 1,036, manholes (cold) - 520, manholes (warm) - 1,379, miscellaneous - 81, people - 275,  street lamps (cold) - 100, and street lamps (warm) - 683. The data is split into train and test subsets at random with 634 and 159 images, respectively, to match an 80-20 division. All classes and both cities are represented in both sets. All classes and both cities are represented in both splits.

For the annotation of the thermal images, the image processing program VGG Image Annotator from the Visual Geometry Group was used. The urban features were outlined with polygons, placed as closely as possible to but still outside the object in question.

Usage:

Each compressed zip file of the format "XY_##" represents one of the 14 UAS flights. These contain Numpy files (one per image) of the shape (3000, 3750, 4) where the dimensions are [R, G, B, Thermal]. These will be decompressed into the file structure:

images/
└── KA_01/
└── DJI_0_0013_R.npy
└── DJI_0_0031_R.npy
└── ...

The original polygon annotations performed on the 640 x 512 thermal images can be found in the "json_annotations.zip" for reference. The semantic segmentation model itself requires segmentation masks, which can be found in the zip files of the format "XY_##_masks" for each of the UAV flights. These will be decompressed into the file structure:

masks/
└── KA_01/
└── DJI_0_0013_R.npy
└── DJI_0_0031_R.npy
└── ...

For the related publication, the train and test were used according the "train.txt" and "test.txt" files, generated through a random split to 80-20. The files contain paths in the form "XY_##/DJI_#_####_R.npy" to point towards the images used for the specific set.