How Much Does Architecture Really Matter? A Statistical and Cost-Aware Benchmark of Deep Learning Models for Sentinel-1 Water Segmentation

Complete, self-contained analysis package accompanying the manuscript “How Much Does Architecture Really Matter? A Statistical and Cost-Aware Benchmark of Deep Learning Models for Sentinel-1 Water Segmentation” (submitted for peer review, 2026). The study benchmarks nine convolutional encoder–decoder architectures for segmenting water from Sentinel-1 Synthetic Aperture Radar imagery of the Poechos reservoir, northern Peru. No deep-learning model was trained here: the study reuses 45 pre-trained checkpoints (three decoders — U-Net, U-Net++ and Feature Pyramid Network — crossed with three encoders — ResNet50, EfficientNet-B7 and Inception-v4 — each as a five-fold ensemble) developed by a companion reservoir-monitoring study, and interrogates them with a purely analytical protocol. Research questions Are the accuracy differences among the nine models statistically real, and is the systematic signal driven by the encoder or by the decoder? Does the internal cross-validation ranking predict performance on independent external data? Once accuracy is accounted for, how large is the spread in computational cost, and which models are Pareto-optimal? Does a statistical tie translate into interchangeable water masks? Main findings Across five cross-validation folds the nine models span only 0.53 percentage points of mean Intersection-over-Union. A two-way analysis of variance blocked by fold shows that the data partition explains 98.0% of the accuracy variance, the encoder a small but significant 0.71% (p = 0.0002), and the decoder architecture a non-significant 0.07% (p = 0.334). A Friedman test confirms a reproducible but marginal ordering, while Wilcoxon–Holm post-hoc tests resolve no individual pair. The internal ranking is essentially uncorrelated with the independent external-test ranking (Spearman ρ = −0.03; bootstrap 95% CI [−0.35, +0.68]); EfficientNet-B7 is the only backbone ranked first in both regimes. Computational cost varies five-fold in inference time (76–369 ms per patch) and seven-fold in floating-point operations (15.6–114.9 GFLOPs); all three Pareto-optimal models use Feature-Pyramid-Network decoders, and every U-Net and U-Net++ variant is dominated. A faithful replication of the companion study’s threshold-calibration pipeline reproduces its published optimum (0.64) exactly; per-model calibration adds only 0.4–1.2 points of pooled IoU. On a held-out scene, the best- and worst-ranked calibrated models disagree on only 0.27% of pixels, their water masks overlapping at an IoU of 98.2% — the statistical tie is also a practical one. Conclusion. When segmentation accuracy is statistically indistinguishable, model selection should be governed by computational cost. The protocol is transferable to other applied segmentation benchmarks. Contents scripts/ — all analysis and figure-generation scripts (Python, 13 files). figures/ — the eight manuscript figures as 300-dpi TIFF images, plus the editable draw.io source of the workflow diagram. derived_tables/ — all result tables: consolidated cross-validation metrics, ANOVA and Friedman outputs, rank-correlation results, computational-cost profile, Pareto set, per-model threshold-calibration curves and the mask-agreement statistics. base_study_data/ — the lightweight inputs reused from the companion study: per-fold cross-validation metric records, ensemble results and configurations for the nine models, and three held-out Sentinel-1 test scenes with their PlanetScope ground-truth masks. What is and is not reused. This package reuses only the artefacts of the companion study’s segmentation stage: the 45 trained checkpoints, their per-fold validation records, and the held-out test scenes with their ground-truth masks. The SWOT altimetry, the stochastic coupling, the water-area time series and the elevation–area–volume curve are outputs of the companion study and are not used in this analysis; the two spreadsheets under base_study_data/spreadsheets/ are included for provenance and context only. No new imagery, model or map was generated. Data provenance. All benchmarked models, cross-validation metrics and Sentinel-1 patches originate from the companion study, openly archived at https://doi.org/10.5281/zenodo.21253273. The 45 model checkpoints (~8.7 GB) and the training patches are not duplicated here, to avoid a redundant multi-gigabyte re-upload: they are required only to reproduce the computational-cost, calibration and mask-agreement analyses, while the statistical benchmark is fully reproducible from the cross-validation metric files provided in this deposit. See base_study_data/MODEL_WEIGHTS_NOTE.txt. License: Creative Commons Attribution 4.0 International (CC BY 4.0).