Coherence-based prediction of multi-temporal InSAR measurement availability for infrastructure monitoring - research data

This dataset contains the input rasters, processed layers, model outputs, and figures supporting the paper “Coherence-Based Prediction of Multi-Temporal InSAR Measurement Availability for Infrastructure Monitoring”, published in the IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 17, 16392–16410 (2024). DOI: 10.1109/JSTARS.2024.3449688. The study trains a 2-channel U-Net to predict European Ground Motion Service (EGMS) persistent-scatterer (PS) density from Sentinel-1 seasonal coherence and an OpenStreetMap infrastructure map. The training and test area is the Netherlands and the bordering strip of Germany and Belgium, longitude 3–8°E and latitude 50–54°N (20 tiles of 1°×1°: N51–N54, E003–E007). A transfer test uses a crop over Rome. An example prediction is given for one tile in Silesia (N51E018). The Netherlands analysis grid is 6000 × 4800 pixels, 3 arcseconds, EPSG:4326, origin (3°E, 54°N). Coherence is stored as unsigned integers and is divided by 1000 in the paper script. The archive is organised as input/, input_Italy/, input_plots/, input_predict_region/, EGMS_data_for_paper_plots/, results_train/, results_test/, results_test_rome/, results_predict_region/, and results_paper/. Set data_root in the companion code to this directory. Paths match config/paths.example.yaml. References to step numbers below follow that pipeline (steps 1–7). Most users can skip steps 1–3 and train or plot from the rasters already in the archive. Summer coherence tiles can also be downloaded again from the public Sentinel-1 global coherence bucket. The 2-channel paper checkpoint is not in this archive. The test and predict scripts download it from Hugging Face (dominika-malinowska/ps_predictions, file seed_54321_20231214-203338.ckpt) unless a local checkpoint is set. With the 20 Netherlands tiles, seed 54321, and that checkpoint, the test loader has 97 batches. The RGB basemap used as the left-hand panel of Figs 6, 12, 13, 14, and 15a–b is not included. Any true-colour GeoTIFF can be placed in input/rgb_base_map/ and input_Italy/rgb_base_map/. It does not have to be Google imagery. The published JPEGs are in the archive, so those figures do not need to be redrawn. Test error and the prediction GeoTIFFs do not use this imagery. Netherlands inputs (input/) Coherence (coherence_summer_NL/summer_vv_rho/): summer VV coherence-decay parameter ρ for December 2019–November 2020, as 20 tiles of 1°×1° plus the mosaic summer_vv_rho.tif. Train and test must use the tiles/ directory. A single mosaic changes the 70/15/15 split and the test error. Source: Kellndorfer et al., Global seasonal Sentinel-1 interferometric coherence and backscatter data set, NASA ASF DAAC, DOI: 10.5067/8W33RRS6S2RV; paper DOI: 10.1038/s41597-022-01189-6. Public bucket s3://sentinel-1-global-coherence-earthbigdata/data/tiles. OpenStreetMap (OSM/): OSM_combined.tif is the infrastructure raster used for training. Class codes are rail 0.86, building 0.88, motorway 0.73, trunk 0.55, primary 0.59, and secondary 0.59. Primary and secondary share one code, so they cannot be separated from this file. OSM/separate/ holds one binary raster per class (buildings, railway, highway, highway_motorway, highway_trunk, highway_primary, highway_secondary) for the paper figures. OSM/osm_source/ holds some of the Geofabrik extracts used to build the map: netherlands-latest.osm.pbf, drenthe-latest.osm.pbf, and nordrhein-westfalen-latest.osm.pbf, downloaded 26 October 2023 (src.txt). © OpenStreetMap contributors, ODbL. A new run of step 2 writes a timestamped mosaic and does not replace OSM_combined.tif. EGMS (EGMS/): egms_point_density.tif is the PS count per 3-arcsecond pixel, used as the training target. It was built in QGIS from EGMS L2b points. egms-archive-links_example.txt is an example of the portal link list for this area of interest. egms-archive-links_example_filtered.txt is the orbit-filtered list written by prepare_egms.ipynb. The L2b shapefiles themselves are not included. Source: European Ground Motion Service, Copernicus Land Monitoring Service (https://egms.land.copernicus.eu/). NDVI (NDVI/): summer and winter mean and standard deviation (mean_ndvi_S2_su_nan_filled_resized.tif, mean_ndvi_S2_wi_nan_filled_resized.tif, std_ndvi_S2_su_nan_filled_resized.tif, std_ndvi_S2_wi_nan_filled_resized.tif). Built from Sentinel-2 in Google Earth Engine (NDVI from 10 m NIR and red, scenes with less than 50% cloud cover in the same seasonal windows as the coherence product, clouds and shadows masked, then mean and standard deviation). The rasters were later gap-filled and resampled to the 3-arcsecond grid. That resampling step and the Earth Engine script are not in the code repository, so to recreate the results, please use these GeoTIFFs. One-channel checkpoint (nn_models/one_input_channel/seed_54321_20231218-105437.ckpt): comparison model for Fig. 13. Not published on Hugging Face. Each analysis raster also has a .npy file with the same name. These are the flattened grids (28,800,000 values) that step 7 loads when they are present. They can be rebuilt from the GeoTIFFs. Seasonal coherence for the paper figures (input_plots/) Spring, fall, and winter VV ρ mosaics, each with the same 20 tiles and a .npy twin: spring_vv_rho.tif, fall_vv_rho.tif, winter_vv_rho.tif. Step 7 reads the mosaics. The same product and source as the summer coherence. If a mosaic is missing, the paper script downloads that season with prepare_coherence. Rome transfer test (input_Italy/) Crop covering about 12.10–12.95°E and 41.62–42.05°N. coh_Rome_cropped/: N42E012_summer_vv_rho.tif and N43E012_summer_vv_rho.tif, summer VV ρ cropped to that extent. OSM_Rome_combined/Rome_combined.tif: infrastructure raster on the same grid. EGMS_mask_Rome/egms_point_density_Rome.tif: EGMS PS-density raster. Silesia example (input_predict_region/) OSM/slaskie-latest.osm.pbf, a Geofabrik extract of the Silesian voivodeship (file date 11 June 2024). Step 6 clips this file to longitude 18.53–18.90°E and latitude 50.05–50.50°N, downloads the summer coherence tile N51E018, and writes the prediction. © OpenStreetMap contributors, ODbL. EGMS points for Figs 15c–d (EGMS_data_for_paper_plots/) underestimation.pkl and overestimation.pkl are GeoPandas frames of EGMS PS points for the two map insets. The source shapefiles are not included because of their size. Step 7 reads the pickle when it is present. Outputs results_train/fig2.jpg: train, validation, and test chip map (step 4 notebook). results_test/merged_prediction_NL.tif: Netherlands PS-density prediction from the 2-channel model. Pixels outside the test split are 255. Step 5 writes a timestamped copy and does not replace this file. results_test/plots/review_NL_2channel/: Figs 6a–d, 13a–b (2-channel), 14a–c, and 15a–b. Each panel is stored twice, as fig*.jpg and as model_pred_b{batch}_{patch}.jpg. results_test/plots/review_NL_1channel/: Fig. 13a–b from the 1-channel model (fig13a_1in.jpg, fig13b_1in.jpg). results_test/plots/notebook_test_pred/: two example chips from the test notebook, not numbered paper figures. results_test_rome/merged_prediction_Rome.tif: Rome prediction. A new run writes a timestamped file beside it. results_test_rome/plots/review_Rome_2channel/: Figs 12a–b. results_test_rome/plots/notebook_rome/: one example chip from the notebook. results_predict_region/N51E018/: example step-6 run. N51E018_PS_pred.tif is the prediction. coh/N51E018_summer_vv_rho.tif is the downloaded coherence tile, infr_map/N51E018_infr_map.tif is the rasterised infrastructure, and clipped_N51_E018.o5m is the clipped OpenStreetMap file. The .o5m is recreated by the script from the Geofabrik extract. results_paper/: Fig. 1 (flowchart), Figs 3a–b, 4a–b, 5a–f, 7a–b, 8, 9, 10a–f, 11a–l, and 15c–d. Figs 3a, 4a, 5a–f, 7a–b, 8, 9, 11a–l, and 15c–d are written by step 7. Figs 1, 3b, 4b, and 10a–f were drawn outside the repository (a flowchart, and QGIS maps of the infrastructure raster, the EGMS density raster, the coherence mosaics, and the NDVI rasters). A new run of step 7 writes results_paper_YYYYMMDD_HHMMSS/ and does not replace these files. Data formats Raster data: GeoTIFF (.tif), EPSG:4326 Flattened analysis arrays: NumPy (.npy) Vector extracts: OpenStreetMap Protocolbuffer Binary (.osm.pbf) and one clipped .o5m Tabular data: pickle (.pkl), plain text link lists Model weights: PyTorch Lightning checkpoint (.ckpt) Figures: PNG and JPEG Source note: input/OSM/osm_source/src.txt Related publication Dataset supports: Malinowska, D., Milillo, P., Briggs, K., Reale, C. & Giardina, G. Coherence-based prediction of multi-temporal InSAR measurement availability for infrastructure monitoring. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 17, 16392–16410 (2024). DOI: 10.1109/JSTARS.2024.3449688. Code repository: https://github.com/SafeStruct/PredictPS 2-channel checkpoint: https://huggingface.co/dominika-malinowska/ps_predictions Third-party sources (please cite when reusing) Kellndorfer, J. et al. Global seasonal Sentinel-1 interferometric coherence and backscatter data set. Sci. Data (2022). DOI: 10.1038/s41597-022-01189-6. Data: NASA ASF DAAC, DOI: 10.5067/8W33RRS6S2RV. Subset used: seasonal VV ρ (summer, spring, fall, winter), tiles over the Netherlands (N51–N54, E003–E007), Rome (N42E012, N43E012), and Silesia (N51E018). OpenStreetMap contributors. OpenStreetMap. https://www.openstreetmap.org (ODbL). Geofabrik extracts: Netherlands, Drenthe, and Nordrhein-Westfalen, 26 October 2023; Silesian voivodeship, 11 June 2024. https://download.geofabrik.de/ European Ground Motion Service (EGMS), Copernicus Land Monitoring Service. https://egms.land.copernicus.eu/. L2b PS points underlying the Netherlands and Rome density rasters and the Fig. 15 pickles. Sentinel-2 MSI, Copernicus, via Google Earth Engine, for the NDVI mean and standard-deviation rasters.