Degradation of ADS-B Surveillance Integrity: Temporal Clustering and Anomaly Persistence

Data License: Creative Commons Attribution 4.0 International (CC BY 4.0) Code License: MIT Overview Welcome to the replication dataset for the study "Degradation of ADS-B Surveillance Continuity: Temporal Clustering and Anomaly Persistence". This repository contains the complete open-science codebase, raw extraction shards, pre-computed survival matrices, and statistical artifacts required to fully reproduce the longitudinal analysis of 2025 global ADS-B tracking stability. Extracting from a gross volume of 246.7 billion OpenSky Network ADS-B messages logged across the 2025 calendar year, this study analyzes 229.0 billion filtered, airborne state vectors. The customized, out-of-core computational pipeline executes scrambled-temporal permutation testing ($\mathcal{W}=\{3,5,7,10,15\}$), time-to-recurrence Kaplan-Meier survival modeling, and exposure-adjusted Cox proportional hazards regression to quantify longitudinal surveillance continuity across 233,859 unique aircraft trajectories. Quick Start & Reproducibility To replicate the temporal clustering and survival modeling pipeline locally: Install dependencies: pip install -r requirements.txt Execute Stage 1: Run CODE/stage1_trino_extraction.py on the OpenSky Trino cluster to extract sequential data (Note: Requires OpenSky credentials). Execute Stage 2: Run CODE/stage2_data_generation.py to execute the Numba JIT-compiled permutation tests and build the survival cohorts. Execute Stage 3: Run CODE/stage3_figure_generation.py to ingest the Stage 2 .parquet files and rapidly re-render all manuscript PDFs and CSV tables. Directory Structure . ├── README.md ├── requirements.txt ├── CODE/ │ ├── config.py │ ├── extract_manuscript_data.py │ ├── stage1_trino_extraction.py │ ├── stage2_data_generation.py │ └── stage3_figure_generation.py ├── DATA/ │ ├── H2_enhanced_results.parquet │ └── extractions/ │ ├── Dataset_2A_aggregated_2025XXXX.PARQUET.zip │ ├── Dataset_2B_daily_spans_2025XXXX.PARQUET.zip │ ├── Dataset_2B_ledger_2025XXXX.PARQUET.zip │ └── Q1_diagnostic_stats_2025XXXX.CSV.zip ├── LOGS/ │ ├── stage1_trino_extraction.log │ ├── stage2_data_generation.log │ └── stage3_figure_generation.log └── RESULTS/ ├── data_permutation_validation.csv ├── df_gaps_raw.parquet ├── df_ledger.parquet ├── fig_censoring_sensitivity_data.csv ├── fig_continental_sensitivity_data.csv ├── fig_exposure_stratified_survival_data.csv ├── fig_stratified_survival_data.csv ├── gap_stats.parquet ├── manuscript_fill_values.json ├── Master_Survival_Cohort.parquet ├── model_cox_coefficients.csv ├── model_schoenfeld_residuals.txt ├── tab_block_sensitivity.csv ├── tab_censoring_demographics.csv ├── tab_per_aircraft_clustering.csv └── figures/ ├── Figure_1_pipeline_architecture.pdf ├── Figure_2_stratified_recurrence.pdf ├── Figure_3_cox_exposure_stratified.pdf ├── Figure_S1_censoring_sensitivity.pdf ├── Figure_S2_dropout_sensitivity.pdf ├── Figure_S3_penalizer_sensitivity.pdf └── Figure_S4_continental_sensitivity.pdf File Manifest & Data Dictionary 1. Root Files README.md: This documentation file. requirements.txt: Standardized list of Python dependencies (e.g., numba, lifelines, pyarrow) to ensure environment reproducibility. 2. CODE/ (The Computational Pipeline) config.py: The central configuration module. Contains the exact kinematic anomaly thresholds (e.g., speed $> 6670.3$ m/s), memory-guard limits, and temporal window size parameters used universally across all pipeline stages. extract_manuscript_data.py: A parsing utility that reads the logs, CSVs, and Parquet files to dynamically extract and format the macro-statistics (like total flight hours and fleet sizes) needed for the manuscript text. stage1_trino_extraction.py: The distributed database extraction script. Executes native SQL window functions on the OpenSky Trino cluster to download and pair daily trajectory trajectories. stage2_data_generation.py: The core data science engine. Runs the out-of-core scrambled-temporal permutation tests (via Numba JIT compilation) and aggregates raw trajectories into discrete monthly survival datasets. stage3_figure_generation.py: The visualization engine. Ingests Stage 2 data to compute the Cox Proportional Hazards models, log-rank tests, and renders the publication-ready PDF figures and CSV tables. 3. DATA/ (Inputs and Intermediates) H2_enhanced_results.parquet: The pre-computed master results matrix from the Stage 2 permutation tests, containing Z-statistics, O/E ratios, and FDR-corrected p-values for all evaluated window scales. extractions/Dataset_2A_aggregated_*.PARQUET.zip: Raw binary anomaly vectors used for independent multi-scale testing. extractions/Dataset_2B_daily_spans_*.PARQUET.zip: Daily observation boundaries used to track network dropouts. extractions/Dataset_2B_ledger_*.PARQUET.zip: The severity-classified anomaly ledger. extractions/Q1_diagnostic_stats_*.CSV.zip: Daily data quality reports (e.g., duplicate message rates). 4. LOGS/ stage1_trino_extraction.log: Extraction timing and data volume logs. stage2_data_generation.log: Permutation tracking and cohort construction logs. stage3_figure_generation.log: Modeling outputs, including the in-sample Cox Model C-Index. 5. RESULTS/ (Final Datasets, Tables, and Models) Master_Survival_Cohort.parquet: The core discrete life-table dataset. It anchors $T_0$ to individual observation inception to prevent immortal time bias and provides the basis for all Kaplan-Meier and Cox models. df_ledger.parquet: Reconciled monthly ledger mapping raw anomalies to their specific physical Severity Classes. df_gaps_raw.parquet: Consolidated raw daily observation spans. gap_stats.parquet: Processed inter-observation gaps used to enforce 30-day network dropout right-censoring. data_permutation_validation.csv: Results of the empirical validation subset confirming high classification concordance between $N=200$ and $N=10,000$ permutations. model_cox_coefficients.csv: Complete Cox Proportional Hazards regression outputs ($\beta$ coefficients, hazard ratios, $p$-values). model_schoenfeld_residuals.txt: Outputs from the scaled Schoenfeld residuals test evaluating the proportional hazards assumption. manuscript_fill_values.json: An auto-generated JSON key-value store containing all dynamic scalars referenced directly in the manuscript body. 6. RESULTS/ (Manuscript Tables & Curve Coordinates) tab_per_aircraft_clustering.csv: Per-aircraft temporal clustering statistics across all window scales. tab_block_sensitivity.csv: Comparison of significance rates evaluating the uniform-null versus the stricter block-preserving null model. tab_censoring_demographics.csv: Demographic profiles of the resulting survival cohorts by censoring status. fig_stratified_survival_data.csv: Coordinate data for Figure 2 (Severity Class stratification). fig_exposure_stratified_survival_data.csv: Coordinate data for Figure 3 (Flight hours tertile stratification). fig_censoring_sensitivity_data.csv: Coordinate data for Figure S1 (Complete cases vs baseline). fig_continental_sensitivity_data.csv: Coordinate data for Figure S4 (Continental-only subset). 7. RESULTS/figures/ (Publication Graphics) Figure_1_pipeline_architecture.pdf: Methodological flowchart. Figure_2_stratified_recurrence.pdf: Time-to-first anomaly survival functions by physical Error Mode. Figure_3_cox_exposure_stratified.pdf: Exposure-adjusted survival hazard stratifications. Figure_S1_censoring_sensitivity.pdf: Sensitivity impact of network dropouts vs administrative censoring. Figure_S2_dropout_sensitivity.pdf: Stability check across 15, 30, 45, and 60-day network dropout thresholds. Figure_S3_penalizer_sensitivity.pdf: Model coefficient stability versus L2 Ridge penalizer variance. Figure_S4_continental_sensitivity.pdf: Sensitivity control isolating oceanic line-of-sight dropouts. How to Cite If you utilize this dataset, pipeline architecture, or empirical baseline in your research, please cite the accompanying manuscript: Pik, E. (2026). Degradation of ADS-B Surveillance Continuity: Temporal Clustering and Anomaly Persistence. Aerospace Science and Technology (Under Review). Dataset DOI: 10.5281/zenodo.19698275