Evaluating In-Domain Transfer Learning for Multispectral Land Cover Mapping in Low-Data Regimes - Supplementary material

This is supplementary material to the journal paper, "Evaluating In-Domain Transfer Learning for Multispectral Land Cover Mapping in Low-Data Regimes", article submitted - waiting for decision. There is also another Zenodo deposit acting as supplementary part II (DOI: 10.5281/zenodo.17659180) containing additional OpenStreetMap (OSM[*]) files. These are PBF and Geopackage files orginally downloaded from Geofabrik, and similar files for ocean polygons from Natural Earth, and these are only used in the making of OSM class TIFFs to fullfill OSM classes where OSMnx cannot. These additional OSM files are not used in the training of the models. The full size of this deposit is ~175 GB, and the other deposit has size ~49 GB. --------------------------- This deposit consists of three main ZIP files: Data_split.zip: Note: The full Data_split.zip is split into multiple ZIP-parts ending with Data_split.z* . This is due to the very large full size of the dataset and the restricted uploading capacity of Zenodo. Dataset reconstruction instructions are found in README_Data_Reconstruction.txt. The full dataset is split into a total of 16 parts: 15 Data_split.z[01->15] (10 GB per file), and Dataset_split.zip (~7.21 GB) is the final chunkand the main ZIP descriptor. Contents of the full Data_split.zip: Train_data/ - Folder containing the optical Sentinel-2 L2A TIFFS and the OSM class TIFFS for training the models for land cover and land use classification. Test_data/ - Folder containing the optical Sentinel-2 L2A TIFFs and the OSM class TIFFS for evaluating the trained models. Dataset_location_map.png - Map of the training and test image locations. Training images are shown as blue circles, and test images are shown as red triangles. README.txt - Includes some extra information about the data to be aware of. L2A_[train/test]_[ID/name]s.txt - Contain the names and IDs of the L2A images for training and evaluation used to download from the Copernicus Browser through a Python API[*]. (download_images.py in Source_code.zip) Source_code.zip: Model_ckpts/ - Folder containing the Tensorboard logged model checkpoints of the three models trained for each of the five training data scenarios, saved at their optimal training loss curve. An additional checkpoint file indicates the pretrained Terrain model ckpt containing the in-domain pretraiend weights for this model. download_images.py - Python API script for downloading the Sentinel-2 L2A SAFE-files from the Copernicus Browser using the L2A image and ID files from Data.zip. [Requires a Copernicus Browser profile] OSM_scripts_requirements_conda.yaml - Necessary requirements and versions needed to create an anaconda environment, which can run the OSM_osmnx.py and Mask_osm_clouds.py files.[****] OSM-osmnx.py - Reads the downloaded L2A SAFE-files, applies appropriate processing steps to produce 12 band L2A TIFFs. It is also used to produce a TIFF file for each L2A image, containing OSM classes using OSMnx, and some additional external OSM files[***]. Mask_OSM_clouds.tif - Uses the Sentinel-2 scene classification layer (SCL) in the raw L2A SAFE-files to mask out unwanted cloud pixels from the optical images by producing new OSM_cloudmask TIFFs by assigning cloud related pixels in the NoData OSM class. Compress_[L2A/OSM]_tiffs.sh - Linux bash scripts for further compressing of the produced L2A and OSM TIFFS for lower memory consumption and faster reading during the training of the models. Requirements.yaml - Contain the necessary requirements and versions needed to create the environment for running the model training and evaluation scripts.[****] [train/test]_dataset.yaml - Contain the paths to the training and test datasets (L2A and OSM TIFFs), and processing information needed like image chip window sizes and NoData thresholds. The remaining 7 Python scripts are the main scripts used to preprocess the images, and training and evaluate the models using PyTorch Lightning modules, Tensorboard logging, GPU-based training, among others: [ClassBlocks.py , Eval_network.py , Functions.py , Network.py , Read_images.py , Solver.py , Train_network.py] Results.zip: The folders RandInit/, ImageNet/ and Terrain/ contain the Tensorboard files produced during evaluation of the trained models, split into folders for each of the training data scenarios at 100%, 30%, 10%, 3% and 1% of the training data used to train the models. RandInit_statistics.xlsx - Contains evaluation metrics produced during the evaluation of the 7 different RandInit models trained at 100% training data to produce statistics about the randomly initialized models. Analyse_RandInit_statistics.py - Reads the RandInit_statistics.xlsx file and produces a summary of the average statistics of the metrics across the models, like the mean, standard deviation, 95% confidence interval, minimum and maximum values. It also produces classwise averages across the models for several metrics like the mean and standard deviation.' README.txt: Should be more or less a copy of the Zenodo deposit description. README_Data_Reconstruction.txt: Gives an instruction on how to assemble and extract the Data_split-multi-part files into one ZIP, and then extracting the full dataset contents into a folder (ready for use). --------------------------- Note:[*] OpenStreetMap is licensed under the Open Database License (DObL) (openstreetmap.org/copyright).[**] L2A raw SAFE files and intermediate files are not included. Only final L2A and OSM TIFF files for training and evaluating the models.[***] The two OSM processing files are specifically designed to handle some of the Sentinel-2 images where there are some difficulties with getting the OSM classes regularly. Some images require specific PBF or Geopackage files to get the classes. And some has an ocean class addition from Natural Earth to extend OSM water near ocean. These additional files are found at the second linked Zenodo deposit.[****] Current libraries and versions working at the time of experiments.