Degradation of Automatic Dependent Surveillance-Broadcast Integrity: Temporal Clustering and Anomaly Persistence

Degradation of Automatic Dependent Surveillance-Broadcast Integrity: Temporal Clustering and Anomaly Persistence Data License: Creative Commons Attribution 4.0 International (CC BY 4.0) Code License: MIT Overview Welcome to the replication dataset for the study "Degradation of Automatic Dependent Surveillance-Broadcast Integrity: Temporal Clustering and Anomaly Persistence". This repository contains the complete open-science codebase, the daily extraction shards, the pre-computed survival matrices, and the statistical artifacts required to fully reproduce the longitudinal analysis of 2025 global ADS-B tracking integrity. Extracting from a gross volume of 246.7 billion OpenSky Network ADS-B messages logged across the 2025 calendar year, this study analyzes 229.0 billion filtered, airborne state vectors. The out-of-core computational pipeline executes scrambled-temporal permutation testing ($\mathcal{W}={3,5,7,10,15}$), day-resolution time-to-first-anomaly Kaplan-Meier survival modeling under two explicit censoring conventions, exposure-adjusted Cox proportional hazards regression, and an Andersen-Gill recurrent-event model, quantifying longitudinal surveillance integrity across 233,859 unique aircraft trajectories. Version 2 rebuilds the survival analysis at day resolution and corrects five implementation defects found in the Version 1 survival scripts during peer-review revision. The clustering pipeline is unchanged, and a from-scratch re-execution reproduced its aggregates bit-identically. See CHANGELOG.md for the full list. Superseded Version 1 scripts and outputs remain available under Version 1 of this record. Quick Start & Reproducibility To replicate the pipeline locally: Install dependencies: pip install -r requirements.txt (Python 3.10+; the pipeline auto-throttles worker parallelism and runs on machines with 8 GB of RAM). Execute the pipeline. All stages below run entirely from the daily shards in DATA/extractions/; the shards are the Stage 1 extraction outputs, so no OpenSky credentials are required. CODE/stage2_data_generation.py: The Version 1 clustering engine (Numba JIT permutation tests). Optional: its results are pre-computed in DATA/H2_enhanced_results.parquet. CODE/stage2b_daygrain_survival_generation.py: Builds the day-grain survival cohort from the daily ledgers (gap-anchored primary convention; event-priority sensitivity). CODE/stage2c_exposure_recurrence_models.py: Cox time-to-first model, Andersen-Gill recurrent-event model, placebo suite, and ridge-penalizer sweep. CODE/stage2d_null_model_diagnostics.py: Injected-independence null calibration and index-space diagnostics. CODE/stage2b_s2_threshold_sweep.py: Dropout-censoring threshold sweep (15/30/45/60 days) with an internal identity gate against the main cohort. CODE/stage3b_daygrain_metrics_export.py: Exports every manuscript metric and figure-ready curve. CODE/stage3_figure_generation.py: Re-renders the manuscript figure PDFs with an internal integrity gate. Independent re-execution of this package reproduced the manuscript's block-sensitivity aggregates and the permutation-validation agreement bit-identically. Directory Structure . ├── README.md ├── CHANGELOG.md ├── requirements.txt ├── CODE/ │ ├── config.py │ ├── extract_manuscript_data_v2.py │ ├── stage2_data_generation.py │ ├── stage2b_daygrain_survival_generation.py │ ├── stage2b_s2_threshold_sweep.py │ ├── stage2c_exposure_recurrence_models.py │ ├── stage2d_null_model_diagnostics.py │ ├── stage3b_daygrain_metrics_export.py │ └── stage3_figure_generation.py ├── DATA/ │ ├── H2_enhanced_results.parquet │ └── extractions/ │ ├── Dataset_2A_aggregated_20250101..20251231.parquet (365 daily shards) │ ├── Dataset_2B_ledger_20250101..20251231.parquet (365 daily shards) │ └── Q1_diagnostic_stats_20250101..20251231.csv (365 daily reports) ├── LOGS/ │ ├── stage2_data_generation.log │ ├── stage2b_daygrain_survival.log │ ├── stage2b_s2_threshold_sweep.log │ ├── stage2c_exposure_recurrence.log │ ├── stage2d_null_diagnostics.log │ ├── stage3b_daygrain_metrics.log │ ├── stage3_figure_generation.log │ └── stage3_figure_generation_v13.log └── RESULTS/ ├── Master_Survival_Cohort_DayGrain.parquet ├── Master_Survival_Cohort_DayGrain_d15.parquet ├── Master_Survival_Cohort_DayGrain_d45.parquet ├── Master_Survival_Cohort_DayGrain_d60.parquet ├── ledger_daygrain_monthly.parquet ├── df_ledger.parquet ├── aux_max_internal_gap_days.parquet ├── model_cox_daygrain_coefficients.csv ├── model_cox_daygrain_month1_excluded.csv ├── model_ag_recurrent_coefficients.csv ├── model_penalizer_sweep_daygrain.csv ├── model_placebo_null_distribution.csv ├── model_placebo_summary.json ├── model_schoenfeld_daygrain.txt ├── Table_S3_block_preserving_sensitivity_multi_w.csv ├── tab_stratified_clustering_rates.csv ├── tab_censoring_demographics_daygrain.csv ├── tab_logrank_daygrain.csv ├── tab_buffer_share.csv ├── tab_dt_translation.csv ├── tab_window_concordance.csv ├── tab_negative_control.csv ├── tab_replacement_sensitivity.csv ├── data_permutation_validation.csv ├── daygrain_persistence_summary.json ├── Additional_Manuscript_Metrics_DayGrain.json ├── manuscript_fill_values_v2.0.json ├── fig_daygrain_survival_data.csv ├── fig_daygrain_severity_strata_data.csv ├── Figure_2_daygrain_severity_onset_survival.pdf ├── Figure_3_daygrain_exposure_tertiles.pdf ├── Figure_S1_censoring_design_sensitivity.pdf ├── Figure_S2_dropout_threshold_sensitivity.pdf ├── Figure_S3_penalizer_stability.pdf ├── Figure_S4_continental_subset.pdf ├── Figure_S5_null_calibration.pdf └── Figure_S6_block_retention_stratified.pdf File Manifest & Data Dictionary 1. Root Files README.md: This documentation file. CHANGELOG.md: What Version 2 adds and corrects, and what remains unchanged. requirements.txt: Pinned Python dependencies for environment reproducibility. 2. CODE/ (The Computational Pipeline) config.py: Configuration ledger V1.1. All kinematic thresholds (e.g., speed $> 6670.3$ m/s), censoring parameters (30-day dropout threshold, both censoring conventions), penalizer grid, memory-guard limits, window scales, and cryptographic salts in one audited module. extract_manuscript_data_v2.py: Parses the pipeline outputs to extract and format every macro-statistic referenced in the manuscript text. stage2_data_generation.py: The Version 1 clustering engine, retained unchanged. Runs the out-of-core scrambled-temporal permutation tests via Numba JIT compilation; its pre-computed results ship in DATA/H2_enhanced_results.parquet. stage2b_daygrain_survival_generation.py: Reconstructs each aircraft's daily observation calendar from the ledgers and builds the day-grain survival cohort under both censoring conventions. stage2b_s2_threshold_sweep.py: Re-derives the cohort at 15/30/45/60-day dropout thresholds; its internal gate verifies the 30-day pass is row-identical to the main cohort. stage2c_exposure_recurrence_models.py: Fits the exposure-adjusted Cox model, the Andersen-Gill recurrent-event model over the aircraft-month panel, the covariate-permutation placebo suite (100 refits), and the ridge-penalizer sweep. stage2d_null_model_diagnostics.py: The injected-independence calibration study quantifying the deployed test's unconditional false-positive rate, plus buffer-share, timestamp-window, and sampling diagnostics. stage3b_daygrain_metrics_export.py: Exports all survival, persistence, and model metrics plus figure-ready curve coordinates. stage3_figure_generation.py: Renders the publication PDFs; refuses to run if its inputs fail the built-in integrity gate. 3. DATA/ (Inputs; unchanged from Version 1) H2_enhanced_results.parquet: Pre-computed permutation-test results (Z-statistics, O/E ratios, FDR-corrected p-values for all $\mathcal{W}$). extractions/: The 365 daily shards behind the 229.0 billion evaluated state vectors: aggregated anomaly vectors (Dataset_2A_aggregated_*), the severity-classified anomaly ledger (Dataset_2B_ledger_*), and daily data-quality reports (Q1_diagnostic_stats_*). These shards are the direct inputs to the Version 2 day-grain pipeline. 4. LOGS/ Timestamped execution logs providing an audit trail of hardware performance, memory-guard triggers, cohort sizes, and model diagnostics: the Version 1 clustering and figure logs plus the Version 2 day-grain generation, threshold-sweep, modeling, diagnostic, metrics, and figure logs. 5. RESULTS/ (Final Datasets, Tables, Models, and Figures — Version 2) Primary Datasets (.parquet): Master_Survival_Cohort_DayGrain.parquet: The core day-resolution survival dataset (233,859 aircraft). Carries both censoring conventions: gap-anchored (primary; 131,067 events, 56.0%) and event-priority (sensitivity; 164,770 events), with pre-event exposure and observability-coverage covariates. Master_Survival_Cohort_DayGrain_d15/d45/d60.parquet: Threshold-sweep cohorts. ledger_daygrain_monthly.parquet: The aircraft-month panel behind the Andersen-Gill recurrent-event model (1,146,596 aircraft-months). df_ledger.parquet: Reconciled anomaly ledger mapping raw anomalies to their physical Severity Classes. aux_max_internal_gap_days.parquet: Per-aircraft maximum internal observation gap (drives the continental subset). Statistical Models & Validation: model_cox_daygrain_coefficients.csv: Cox time-to-first outputs at $\lambda = 0.1$. model_cox_daygrain_month1_excluded.csv: The first-interval-excluded sensitivity fit. model_ag_recurrent_coefficients.csv: Andersen-Gill recurrent-event outputs (655,588 recurrent cluster-months). model_penalizer_sweep_daygrain.csv: Coefficients across $\lambda \in {0.01, 0.05, 0.1, 0.5, 1.0}$. model_placebo_null_distribution.csv and model_placebo_summary.json: The covariate-permutation placebo suite (100 refits; null coefficient 0.0001, SD 0.0027). model_schoenfeld_daygrain.txt: Scaled Schoenfeld residual tests for the proportional-hazards assumption. data_permutation_validation.csv: N=200 vs. N=10,000 validation (100.0% agreement). Diagnostics & Sensitivity Tables (.csv): Table_S3_block_preserving_sensitivity_multi_w.csv: Per-aircraft block-preserving significance classifications across all window scales. tab_stratified_clustering_rates.csv: Utilization-stratified significance and block-retention rates. tab_censoring_demographics_daygrain.csv: Demographic profiles by survival state under both censoring conventions. tab_logrank_daygrain.csv: Log-rank tests between severity-onset cohorts. tab_buffer_share.csv: Day-boundary buffer share of the index space (mean 0.4% of slots). tab_dt_translation.csv: Inter-message interval percentiles (index-to-seconds translation). tab_window_concordance.csv: Duration-standardized window replication (Jaccard 0.99999 across 18,243,276 aircraft-days). tab_negative_control.csv: Injected-independence false-positive rates by density stratum (0.040 vs. nominal 0.05). tab_replacement_sensitivity.csv: Exact without-replacement re-test of the 500 densest aircraft (zero significance changes). Manuscript Scalars & Curves: Additional_Manuscript_Metrics_DayGrain.json, daygrain_persistence_summary.json, manuscript_fill_values_v2.0.json: Every dynamic scalar referenced in the manuscript body (survival probabilities, persistence endpoints, fleet totals). fig_daygrain_survival_data.csv, fig_daygrain_severity_strata_data.csv: The $X,Y$ coordinates plotted in the survival figures. Publication Graphics (.pdf): Figure_2_daygrain_severity_onset_survival.pdf: Time-to-first clustered anomaly by severity at onset (all 131,067 events classified). Figure_3_daygrain_exposure_tertiles.pdf: Exposure-stratified time-to-first survival. Figure_S1_censoring_design_sensitivity.pdf: Gap-anchored vs. event-priority censoring conventions. Figure_S2_dropout_threshold_sensitivity.pdf: Stability across 15/30/45/60-day dropout thresholds. Figure_S3_penalizer_stability.pdf: Cox coefficient behavior across the ridge-penalizer grid. Figure_S4_continental_subset.pdf: Continuously observed (continental) subset control. Figure_S5_null_calibration.pdf: Null calibration under injected independent anomalies. Figure_S6_block_retention_stratified.pdf: Block-preserving retention among uniform-significant aircraft, by utilization stratum. How to Cite If you use this dataset, pipeline architecture, or empirical baseline in your research, please cite the accompanying manuscript: Pik, E. (2026). Degradation of Automatic Dependent Surveillance-Broadcast Integrity: Temporal Clustering and Anomaly Persistence. Reliability Engineering & System Safety (Under Review, JRESS-D-26-02922). Dataset DOI (all versions): 10.5281/zenodo.19698274 This version (v2): 10.5281/zenodo.21878953