Risk scores and Calibration scores

================================================================ZENODO DATA ARCHIVE DESCRIPTIONSNeurIPS 2026 submission: "A Systematic Analysis of OOD DetectionUnder Representation and Training Paradigm Shifts"================================================================

Three archives accompany the code repository on Zenodo. Oncedownloaded and unzipped at the repository root, the paper'sanalysis pipeline runs unchanged: stats_eval.py regeneratesFigures 1 and 3, projection_*_analysis.py regenerates Appendix Ftables, mantel_analysis.py regenerates Appendix G, and thenc_csf_predictivity/ pipeline regenerates Figure 2 and Appendix Hheatmaps from the parquets in nc_csf_predictivity/outputs/ (orrecomputes them from the unzipped score CSVs to verify).

----------------------------------------------------------------1. scores_calibration.zip   (about 114 MB, ~175 CSV files)----------------------------------------------------------------

Per-checkpoint and per-cell calibration outputs. Computed byStage 4 of the paper's pipeline (calibration_ood.py followed byrecompute_metric.py) on every classifier checkpoint evaluated inthe paper. Source backbones are VGG-13 (Conv) and the fine-tunedViT (ViT); ResNet-18 calibration is in scores_risk_resnet18.zip.Source datasets are CIFAR-10, CIFAR-100, SuperCIFAR-100, andTinyImageNet.

Filename schema:

  calibration_results_<source>_<backbone>.csv      Per-(checkpoint, OOD dataset, hyperparameter) raw      calibration outputs. Includes per-bin reliability data;      these are the largest files in the archive (about 22-32 MB      each for the four Conv sources).

  scores_<metric>_MCD-<flag>_<backbone>_<source>.csv      Per-cell aggregated calibration metric, written by      recompute_metric.py.

  scores_all_<metric>_MCD-<flag>_<backbone>_<source>.csv      Full hyperparameter sweep aggregated per (source, backbone,      MCD), stacked across CSFs and projection variants.

  hyperparameters_results_MCD-<flag>_<backbone>_<source>.csv      Best hyperparameter selected per cell (CSF dimensionality,      DG reward, dropout, etc.) by validation-AUGRC minimization.      Drives the table in Appendix A.

Metrics covered: ECE_L1, ECE_L2, ECE_L1_BOUND, ECE_L2_BOUND, MCE.The L1/L2 BOUND variants are the bounded-debiased ECE estimatorsdiscussed in Appendix A.

MCD flag: MCD-False is the deterministic-inference setting usedthroughout the paper. MCD-True files are Monte-Carlo-dropoutvariants kept for completeness; the body uses deterministicinference only.

Used by paper: Appendix A's hyperparameter selection tables(dropout, Deep Gamblers reward) read from hyperparameters_results_*and the scores_all_* files. ECE / MCE values are not the paper'sheadline metric (AUGRC is) but provide auxiliary calibrationevidence supporting the "validation AUGRC drives selection" storyin Appendix A.

----------------------------------------------------------------2. scores_risk.zip          (about 70 MB, ~280 CSV files)----------------------------------------------------------------

Per-cell risk-coverage and detection-ROC scores for VGG-13 andViT. Computed by Stage 3 (retrieve_scores.py) and Stage 4(recompute_metric.py) of the paper's pipeline on the FD-Shiftspretrained checkpoints. This is the primary input to alltop-clique, projection-filtering, NC-predictor, and Mantelanalyses in the paper.

Filename schema:

  scores_<metric>_MCD-<flag>_<backbone>_<source>.csv      Hyperparameter-marginalized score per (CSF, projection      variant, OOD dataset, paradigm, run) cell.

  scores_<metric>_MCD-<flag>_<backbone>_<source>_fix-config.csv      Same shape but with hyperparameters locked to the per-cell      best (output of recompute_metric.py).

  scores_all_<metric>_MCD-<flag>_<backbone>_<source>.csv      Full hyperparameter sweep stacked across CSFs and projection      variants. THIS IS THE FILE stats_eval.py READS for the      headline rank pipeline.

  scores_all_<metric>_MCD-<flag>_<backbone>_<source>_fix-config.csv      Same sweep with config-locked hyperparameters; used by the      projection-filtering analyses.

Metrics covered:

  AUGRC      Headline ranking metric for top cliques (Section 4.2)             and the predictor's per-cell oracle (Section 4.4).  AURC       Joint ranking metric paired with AUGRC in             --metric-group RC.  AUROC_f    Auxiliary OOD-detection ROC AUC (filtered against             correctly classified ID samples).  FPR@95TPR  Auxiliary detection metric (ROC operating point).  ECE, MCE   Calibration-side metrics; not headline numbers but             produced by the same retrieve_scores pipeline.

Backbones:  Conv = VGG-13 (FD-Shifts CNN)  ViT  = fine-tuned ViT pool

Both have 4 source datasets x 5 seeds x {ConfidNet, DeVries, DeepGamblers} paradigms x dropout x Deep-Gamblers-reward factors asdocumented in Appendix C (280 VGG-13 + 40 ViT checkpoints).

Used by paper:  - stats_eval.py reads scores_all_AUGRC_MCD-False_* to produce    Figures 1, 3 (top-clique panels).  - projection_filtering_analysis.py reads _fix-config variants    for Tables 15, 16 in Appendix F.  - mantel_analysis.py reads scores_all_AUGRC_MCD-False_* for the    Mantel test in Appendix G.  - nc_csf_predictivity/data/build_dataset.py reads    scores_all_AUGRC_MCD-False_*_fix-config.csv to assemble the    per-cell label matrix used by the cross-architecture    predictor.

----------------------------------------------------------------3. scores_risk_resnet18.zip (about 18 MB, ~95 CSV files)----------------------------------------------------------------

Per-cell risk-coverage and detection-ROC scores for the held-outResNet-18 cross-architecture pool. Same generation pipeline asscores_risk.zip but applied to the ResNet-18 checkpoints trainedfor this paper (4 datasets x 3 paradigms with the dropout/rewardfactors, 56 checkpoints in total; see Appendix C).

Filename schema: identical to scores_risk.zip, with twosimplifications:

  - The backbone field is Conv (FD-Shifts uses "Conv" as the CNN    family identifier; the ResNet-18 backbone files inherit this    label, but the directory name scores_risk_resnet18/    distinguishes them from the VGG-13 outputs in    scores_risk.zip).

  - All files carry the _fix-config suffix because ResNet-18 cells    are scored under the per-cell-locked hyperparameter    configurations exported by recompute_metric.py.

Metrics covered: AUGRC, AURC, AUROC_f, FPR@95TPR, ECE, MCE. Eachmetric appears in both the per-cell scores_<metric>_* form and thefull-sweep scores_all_<metric>_* form.

Used by paper:  - The cross-architecture transfer test in Section 4.4 (the    headline regret reductions of 51-84% on near/mid/far). The    predictor is trained on scores_risk/ (VGG-13 cells) and    applied to this ResNet-18 pool to compute per-row set-regret.  - nc_csf_predictivity/data/cliques_resnet18.py and    data/oracle_regret.py read these files to compute the per-cell    oracle CSF and shortlist-regret values that feed    evaluation/regret_by_side_clique_bc.py (Figure 2).  - The worked example in Appendix I (the    "ResNet18 | confidnet | cifar100 | 1 | 0 | 2.2" row,    evaluated on iSUN, mid-OOD) is one row from this archive.