Open Research Data "Changing Climate Extremes Accelerate the Recurrence of Crop Losses Worldwide"

Hybrid Crop Model Code and data for: "Changing Climate Extremes Accelerate the Recurrence of Crop Losses Worldwide" Zenodo DOI:https://doi.org/10.5281/zenodo.20644298 Code license: MIT (see LICENSE) Data license: CC BY-NC 4.0 Contact: Baoying Shan, polimi, [e-mail], [e-mail] Contents What's in this archive Quick start Installation Configuration Reproduction guide Ablation experiments Project structure Paper element to script mapping Data sources Citation What's in this archive The archive is self-contained: code, processed data, trained models, and pre-computed figure source data are all bundled. Raw upstream datasets are not re-hosted; see Data sources for citations. Path What's inside README.md This file. LICENSE MIT license for the code. requirements.txt Python dependency list. config/ Hostname-keyed YAML configs (config.yaml, config_ISIMIP_3b.yaml). src/ All Python source (data preparation, training, prediction, experiments, visualization). data/ Processed feature tables (5 crops, 5 GCMs). See data/README.md and data/data_dictionary.csv. models/ Trained checkpoints: main model, leave-one-year-out, and leave-one-region-out, per crop. results/ Pre-computed figure source data (performance metrics, return-period NetCDFs, extreme-frequency summaries). reference/ IPCC WGI v4 reference-region shapefile used for spatial aggregation. Quick start After installation and configuration, reproduce Fig. 1 (crop-loss return-period maps) from the bundled results: export PYTHONPATH="." python src/visualization/fig1/plot_fig1.py Output figures are written under figures/. For other reproduction paths, see the reproduction guide. Installation Requirements: Python 3.9+, PyTorch 1.10+, PyTorch Geometric 2.2.0+. GPU training needs CUDA 11.3 or newer; CPU-only also works for figure reproduction and small-scale inference. PyTorch and PyTorch Geometric depend on your local CUDA/CPU build and must be installed from their official wheel indexes before the rest. The full dependency list is in requirements.txt. # 1. Create and activate a conda environment conda create -n crop_gat python=3.9 conda activate crop_gat # 2. Install PyTorch (adjust the CUDA tag as needed) pip install torch==1.10.0+cu113 \ -f https://download.pytorch.org/whl/torch_stable.html # 3. Install PyTorch Geometric and its companion wheels. # Pick the URL that matches your torch / CUDA build; see # https://pytorch-geometric.readthedocs.io/en/2.2.0/install/installation.html pip install torch-scatter torch-sparse torch-cluster torch-spline-conv \ -f https://data.pyg.org/whl/torch-1.10.0+cu113.html pip install torch-geometric==2.2.0 # 4. (Recommended) Install the geospatial stack via conda. # cartopy / geopandas / rioxarray pull in GEOS, PROJ, and GDAL, # which conda resolves cleanly. conda install -c conda-forge cartopy geopandas rioxarray # 5. Install the remaining pure-Python dependencies. # pip skips packages already satisfied by conda in step 4. pip install -r requirements.txt Pip-only alternative: skip step 4 and run step 5 directly. pip will then build cartopy, geopandas, and rioxarray from source, which requires GEOS, PROJ, and GDAL to be available on the system. Configuration Edit config/config.yaml and config/config_ISIMIP_3b.yaml so the bundled paths point to your extracted archive. Find your hostname: python -c "import platform; print(platform.node())" Add a block keyed by your hostname under paths in each config: paths: root: YOUR_HOSTNAME: /absolute/path/to/hybrid_crop_model/ data: YOUR_HOSTNAME: /absolute/path/to/hybrid_crop_model/data/ processed_data: # ISIMIP 3b config only YOUR_HOSTNAME: /absolute/path/to/hybrid_crop_model/data/ Set ipcc_regions_path in both configs to the shapefile inside reference/IPCC-WGI-reference-regions-v4/. Reproduction guide The archive supports four levels of reproduction. Levels 1 to 3 run entirely from bundled artifacts; Level 4 additionally requires the raw upstream datasets. Level Goal Time Hardware Needs 1 Reproduce figures Minutes Any laptop results/ (bundled) 2 Re-run predictions Hours GPU recommended models/ + data/processed_ISIMIP_3b/ (bundled) 3 Retrain models ~24 h / crop GPU strongly rec. data/processed/ (bundled) 4 Full pipeline from raw data Weeks HPC recommended Raw upstream datasets (see Data sources) All scripts below are run from the archive root. export PYTHONPATH="." must be set so that the src package resolves; it appears at the top of each block below. Level 1: Reproduce figures export PYTHONPATH="." # Fig 1 + Fig S3: Crop-loss return-period maps python src/visualization/fig1/plot_fig1.py # Fig 2 + Fig S4: Hybrid vs ISIMIP performance python src/visualization/fig2/plot_fig2.py # Fig 3 + Fig S6: Temporal / spatial generalization python src/visualization/fig3/plot_fig3.py # Fig S2: Return-period validation histograms python src/visualization/figs_val_rp/figs_val_rp.py # Fig S5: Spatial partitioning python src/visualization/figs_sp/plot_figs_sp.py # Fig S7: Extreme-frequency change figure python src/visualization/fig_extreme_freq/plot_extreme_freq_change.py Level 2: Re-run predictions export PYTHONPATH="." # Single-scenario smoke test (one crop * one GCM) python src/experiment/experiment_pi_ISIMIP_obs_exs1.py \ --crop mai --GCM_model GFDL-ESM4 # Attribution analysis (Table 1, Tables S1-S3) python src/experiment/experiment_results_analysis.py Note: src/prediction/run_batch_predictions.py is not part of the public quick start; it currently references a missing GPU-specific script and needs correction before public use. Level 3: Re-train models export PYTHONPATH="." export CROP=mai # or: ri1 / soy / swh / wwh # Short smoke test (1 epoch each) python src/training/combined_model.py \ --epochs_anomaly 1 --epochs_trend 1 \ --output_fdr_name smoke_test_combined_model # Full main-model training (~24 h / crop on a single GPU) python src/training/combined_model.py \ --epochs_anomaly 800 --epochs_trend 400 \ --seed 253 \ --batch_size_anomaly 8 --batch_size_trend 100 \ --sam_adaptive # Generalization tests python src/training/time_test_combined.py # Leave-one-year-out python src/training/spatial_test_combined.py # Leave-one-region-out # Compile performance metrics python src/training/model_performance.py Tip: delete any models/{crop}/processed/smoke_test_combined_model_* directory before generating checksums or preparing a Zenodo upload. On HPC (SLURM): sbatch src/training/combined_model_GPU.sh Level 4: Full pipeline from raw data Download the raw datasets listed in Data sources, then run the preparation pipeline: export PYTHONPATH="." python src/data_preparation/generate_spatial_mask.py # Spatial masks python src/data_preparation/main.py # GDHY + ISIMIP 3a (observed climate) python src/data_preparation/main_ISIMIP_3b.py # ISIMIP 3b picontrol Then proceed to Level 3. Ablation experiments Three ablation variants test individual feature-group contributions. Variant What is removed Training script no_extremes_all Extreme-event predictors src/experiment/ablation/combined_model_no_extremes_all.py ablation_no_ISIMIP All ISIMIP features src/experiment/ablation/combined_model_ablation_no_ISIMIP.py ISIMIP_ensemble_only Individual ISIMIP outputs (ensemble kept) src/experiment/ablation/combined_model_ISIMIP_ensemble_only.py Analyze results: export PYTHONPATH="." python src/experiment/ablation/prediction_no_extremes_results_analysis.py python src/experiment/ablation/prediction_ablation_no_ISIMIP_results_analysis.py python src/experiment/ablation/prediction_ISIMIP_ensemble_only_results_analysis.py Project structure hybrid_crop_model/ ├── README.md # This file ├── LICENSE # MIT (code) ├── requirements.txt # Python dependencies │ ├── config/ # Hostname-keyed YAML configs │ ├── config.yaml # ISIMIP 3a config │ └── config_ISIMIP_3b.yaml # ISIMIP 3b config (primary) │ ├── src/ # Source code │ ├── data_preparation/ # Data loading, cleaning, feature engineering │ ├── training/ # Model training + generalization tests │ ├── prediction/ # Inference and batch predictions │ ├── experiment/ # Attribution analysis │ │ └── ablation/ # Ablation studies │ └── visualization/ # Figure generation │ ├── style.py # Shared plotting style │ ├── fig1/ fig2/ fig3/ # Main-text Figs 1-3 │ ├── figs_val_rp/ # Supplementary Fig S2 │ ├── figs_sp/ # Supplementary Fig S5 │ └── fig_extreme_freq/ # Supplementary Fig S7 │ ├── data/ # Processed input tables │ ├── README.md # Data-package README │ ├── data_dictionary.csv # Variable-level documentation │ ├── processed/ # Observed-climate training tables │ │ └── {crop}/csv/vars_aligned.csv # 5 crops: mai, ri1, soy, swh, wwh │ └── processed_ISIMIP_3b/ # picontrol prediction tables │ └── picontrol/{GCM}/{crop}/csv/vars_aligned.csv │ # 5 GCMs x 5 crops │ # GCMs: GFDL-ESM4, IPSL-CM6A-LR, │ # MPI-ESM1-2-HR, MRI-ESM2-0, │ # UKESM1-0-LL │ ├── models/ # Trained checkpoints (5 crops) │ └── {crop}/processed/ │ ├── combined_model_YYYY_MM_DD/ # Main (random split) │ ├── time_test_combined_{crop}_YYYY_MM_DD/ # Leave-one-year-out │ └── spatial_test_combined_regular_grid_all_{crop}_YYYY_MM_DD/ # Leave-one-region-out │ ├── results/ # Pre-computed figure source data │ ├── hybrid_model_performance/ # Fig 2 + Fig S4 performance source data │ │ ├── all_crops_model_performance_normal_test_set.csv │ │ └── global_weighted_avg_by_year.csv │ ├── experiment_pi_ISIMIP_obs_exs/ # Fig 1 + Figs S2-S3 return-period NetCDFs │ │ └── picontrol/{crop}/{crop}_yield_ano_statistics.nc │ └── extreme_freq_change/ # Extreme-frequency summary │ └── extreme_freq_change_summary.csv │ └── reference/ # External reference data └── IPCC-WGI-reference-regions-v4/ # IPCC WGI regions shapefile Paper element to script mapping Paper element Script Fig 1, Crop-loss return-period maps src/visualization/fig1/plot_fig1.py Fig 2, Hybrid vs ISIMIP performance src/visualization/fig2/plot_fig2.py Fig 3, Temporal / spatial generalization src/visualization/fig3/plot_fig3.py Fig S2, Return-period validation histograms src/visualization/figs_val_rp/figs_val_rp.py Fig S3, Return-period maps (ISIMIP ensemble mean) src/visualization/fig1/plot_fig1.py Fig S4, Hybrid vs ISIMIP (supplementary) src/visualization/fig2/plot_fig2.py Fig S5, Spatial partitioning src/visualization/figs_sp/plot_figs_sp.py Fig S6, Generalization test (supplementary) src/visualization/fig3/plot_fig3.py Fig S7, Extreme-frequency change figure src/visualization/fig_extreme_freq/plot_extreme_freq_change.py Table 1 + Tables S1-S3, Attribution analysis src/experiment/experiment_results_analysis.py Data sources Download DOIs below match the manuscript's Data availability section and are needed only for Level 4 (full pipeline from raw data). All processed inputs for Levels 1 to 3 are already bundled under data/. Dataset Download ISIMIP 3a / 3b crop simulations, land-use forcing, soil data https://data.isimip.org/ GDHY v1.2-1.3 (observed yields; Iizumi et al.) 10.1594/PANGAEA.909132 GSWP3-W5E5 reanalysis (extreme-event identification) via ISIMIP (data.isimip.org) Global Aridity Index v3 (Zomer et al.) 10.6084/m9.figshare.7504448.v6 Gridded GDP (Kummu et al.) 10.5061/dryad.dk1j0 IPCC WGI v4 reference regions (Iturbide et al.) 10.5281/zenodo.3998463 SPI / SHI extreme-event identification code 10.5281/zenodo.11397269 Processed feature tables, trained checkpoints, and pre-computed figure source data are already bundled in this archive under data/, models/, and results/ respectively. See data/README.md for per-file details. Citation If you use this code or data, please cite the paper and the archive: Paper citation: to be added upon publication Zenodo archive: https://doi.org/10.5281/zenodo.20644298