Multi-Dimensional Coupling of Kinematic Anomalies in ADS-B Surveillance Telemetry

Multi-Dimensional Coupling of Kinematic Anomalies in ADS-B Surveillance Telemetry Eugene Pik 26 May 2026 Multi-Dimensional Coupling of Kinematic Anomalies in ADS-B Surveillance Telemetry Data License: Creative Commons Attribution 4.0 International (CC BY 4.0) Code License: MIT Dataset DOI: 10.5281/zenodo.20387308 Overview Welcome to the replication dataset for the study "Multi-Dimensional Coupling of Kinematic Anomalies in ADS-B Surveillance Telemetry". This repository contains the complete open-science codebase, raw extraction shards, intermediate analytical products, statistical artifacts, and publication figures required to fully reproduce the conditional-dependence analysis of 2025 global ADS-B kinematic telemetry. The pipeline analyses 117,542,731 aircraft-hour aggregates clustered by 240,001 unique airframes, and 791,862,868 non-overlapping 300-second tumbling windows, all extracted from the OpenSky Network state_vectors_data4 table for calendar year 2025 (1 January through 31 December 2025 UTC). For the year-over-year temporal-anomaly diagnostic, a calendar-aligned 50-day window of Spring 2024 (2 April through 22 May 2024) is re-extracted, yielding an additional 15,733,909 aircraft-hours across 192,747 airframes. A pair-level pilot analysis on 24 sampled hours yields a further 1,176,250,672 consecutive-message pairs across 76,023 airframes, with velocity_data4-level re-extraction exposing the DF17 BDS 0,9 vertical-rate source bit (VrSrc). The pipeline executes rate-normalised per-step anomaly flagging anchored in 14 CFR Part 25 limit load factors, RTCA DO-260B baro-altitude quantisation, and the standard rate of turn; three-tier data-quality stratification by within-hour Δt-coverage; cluster-bootstrap confidence intervals resampling by airframe; an exact within-cluster permutation null distribution preserving per-airframe marginal counts; cluster-robust phase-stratified logistic regression with symmetric-threshold sensitivity; an explicit cluster-robust regression specification controlling for per-hour message density with a DVR × log(total_pairs) interaction; VrSrc-stratified pair-level analysis directly resolving the barometric / GNSS-source decomposition; and cross-references against the NOAA Storm Events Database and the GFZ Potsdam daily geomagnetic and solar indices archive. Quick Start & Reproducibility To replicate the conditional-dependence analysis locally: 1. Install dependencies: pip install -r requirements.txt 2. Execute the pipeline: Stages are decoupled and write intermediate Parquet shards to allow per-stage reruns. – CODE/stage1_master_metadata_extraction_V5_3.py — Builds the enriched aircraft-metadata registry by merging the OpenSky aircraft database (providing the base icao24, type-designator, registration, and icaoAircraftClass fields), the FAA Aircraft Registry (for US tail-number built_year), Transport Canada CARS (for Canadian tail-number built_year), and the Rikgale ICAO Type designators (for typecode-to-engine and typecode-to-airframe-category resolution). – CODE/stage2_1_pphysical_extraction_V2_2.py — Aggregates per-step rate-normalised flags to aircraft-hour granularity (produces Dataset 3A). Runs against the OpenSky Trino interface; requires OpenSky credentials. – CODE/stage2_2_window_extraction_V2_2.py — Aggregates per-step flags to 300-second tumbling windows (produces Dataset 3B) and classifies each window's manoeuvring envelope. – CODE/stage2_3_metadata_enrichment_V2_6.py — Merges Dataset 3A with the metadata registry and computes pairwise enrichment, conditional-independence violation ratios, and engine-type universality statistics. – CODE/stage2_4_aerodynamic_masking_regression_V1_2.py — Fits the phase-stratified cluster-robust logistic regression on Dataset 3B. – CODE/phase3_2_*.py through CODE/phase3_9_*.py — Phase 3 robustness analyses (data-quality decomposition, DVR threshold sweep, geographic and seasonal stratification, NACv/NIC/receiver-redundancy stratification, pair-level NACv pilot, Spring 2025 elevation diagnostic, year-over-year Spring 2024 re-extraction, space-weather cross-reference, forest-plot generation, publication figures, VrSrc-stratified pair-level pilot, and log(total_pairs)-controlled cluster-robust regression). Dataset Structure This Zenodo dataset consists of three ZIP archives and four loose files: .├── README.md <- this file├── requirements.txt <- Python dependencies├── LICENSE-CODE-MIT.txt├── LICENSE-DATA-CC-BY-4.0.txt│├── replication_kit.zip <- code + results (~1 MB)│ └── replication_kit/│ ├── CODE/│ │ ├── config.py│ │ ├── stage1_master_metadata_extraction_V5_3.py│ │ ├── stage2_1_pphysical_extraction_V2_2.py│ │ ├── stage2_2_window_extraction_V2_2.py│ │ ├── stage2_3_metadata_enrichment_V2_6.py│ │ ├── stage2_4_aerodynamic_masking_regression_V1_2.py│ │ ├── phase3_2_robustness_analyses.py│ │ ├── phase3_3_integrity_extraction_V2_2.py│ │ ├── phase3_3_nac_stratification.py│ │ ├── phase3_4_spring_diagnostic.py│ │ ├── phase3_4b_spring2024_full_window_V1_1.py│ │ ├── phase3_5_combined_robustness.py│ │ ├── phase3_6_space_weather_V1_2.py│ │ ├── phase3_7_figure_generation_V1_2.py│ │ ├── phase3_7_forest_data_builder.py│ │ ├── phase3_8_vrsrc_pilot.py (NEW: V38 / R1.1)│ │ ├── phase3_9_npair_regression.py (NEW: V38 / R1.2)│ │ ├── phase3_engine_breakdown_audit.py│ │ └── rename_dq_tier_columns.py│ └── RESULTS/│ ├── summaries/ <- JSON summaries from every phase│ ├── manuscript_tables/ <- one CSV per LaTeX tab:* label│ ├── spring2025_diagnostics/ <- d1-d8 + storm-events tables│ ├── space_weather/ <- GFZ Potsdam cross-reference│ ├── permutation_null/ <- 1,000-iteration null distribution│ ├── forest_data/ <- enrichment-forest input│ └── figures/ <- Figures 1-4 PDFs + plot data│├── source_registries.zip <- registries (~187 MB)│ └── source_registries/│ └── aircraft_db/│ ├── FAA/ <- MASTER.txt, ACFTREF.txt│ ├── ICAO/ <- rikgale_icao_types.csv│ ├── OpenSky/ <- 8 monthly snapshots│ └── Transport_Canada/ <- carscurr.txt│└── extractions.zip <- parquet shards (~20 GB) └── extractions/ ├── Dataset_3A_aircraft_hour/ <- 12 monthly shards ├── Dataset_3B_300s_windows/ <- 2 monthly shards (Apr, May) ├── Integrity_NACv_NIC_receiver/ <- 12 monthly shards ├── pphysical_monthly_aggregates/ <- 12 monthly shards (Stage 2.1) ├── Windows_monthly_aggregates/ <- 12 monthly shards (Stage 2.2) ├── Spring_2024_re_extraction/ <- year-over-year diagnostic ├── Seasonal_checkpoint_extracts/ <- spring/summer/autumn/winter v2 └── Aircraft_metadata_registry/ <- enriched_aircraft_metadata.parquet File Manifest & Data Dictionary 1. Root Files • README.md — This documentation file. • requirements.txt — Minimal Python dependency list (Python ≥ 3.9; pandas, numpy, scipy, statsmodels, pyarrow, matplotlib, trino-python-client). • LICENSE-CODE-MIT.txt — MIT license covering all Python source code. • LICENSE-DATA-CC-BY-4.0.txt — Creative Commons Attribution 4.0 covering all data files. 2. replication_kit.zip / CODE/ • config.py — Shared configuration module: pipeline thresholds (DV > 10 m/s², DVR > 8 m/s², DH > 15 °/s), data paths, OpenSky Trino credentials helper, seasonal checkpoints, exclusion lists for known broadcaster pathologies. • stage1_master_metadata_extraction_V5_3.py — Builds the cross-referenced aircraft-metadata registry from four sources: the OpenSky aircraft-database snapshots (base table), the FAA Aircraft Registry, Transport Canada CARS, and the Rikgale ICAO Type designators. • stage2_1_pphysical_extraction_V2_2.py — Aggregates per-step DV/DVR/DH flags to aircraft-hour granularity with the alternate-threshold columns pre-extracted for the DVR sweep. Embeds the production Trino SQL filter WHERE dt_seconds > 0 AND dt_seconds <= 60. • stage2_2_window_extraction_V2_2.py — Produces Dataset 3B; classifies each 300-second window as Climb/Cruise/Descent/Transition. • stage2_3_metadata_enrichment_V2_6.py — Merges metadata into Dataset 3A and computes pairwise enrichment, conditional-independence violations, and engine-type universality. • stage2_4_aerodynamic_masking_regression_V1_2.py — Phase-stratified cluster-robust logistic regression and the rotary-wing saturation grid. • phase3_2_robustness_analyses.py — Data-quality tier decomposition, DVR threshold sweep, seasonal-week robustness, geographic robustness, strict-Fixed-Wing replication, helicopter saturation grid. • phase3_3_integrity_extraction_V2_2.py — Aircraft-hour-level NACv/NIC/receiver-redundancy stratification. Strict-high NACv definition is nacv_velocity_min >= 3. • phase3_3_nac_stratification.py — Pair-level NACv pilot. Pair classification is LEAST(nac, p_nac); high-integrity stratum is NACv ≥ 2. • phase3_4_spring_diagnostic.py — 365-day temporal breakdown identifying the Spring 2025 elevation episode and its UTC diurnal, geographic, airframe, and engine localisation. • phase3_4b_spring2024_full_window_V1_1.py — Re-extraction of the calendar-aligned 50-day Spring 2024 window using the identical SQL pipeline. • phase3_5_combined_robustness.py — Permutation null model, Stage 2.4 symmetric-threshold sensitivity, and NOAA Storm Events cross-reference. • phase3_6_space_weather_V1_2.py — GFZ Potsdam daily geomagnetic and solar indices cross-reference against the daily P(DV) trajectory. • phase3_7_forest_data_builder.py and phase3_7_figure_generation_V1_2.py — Assemble the forest-plot input and generate all four publication figures. • phase3_8_vrsrc_pilot.py (NEW, V38) — VrSrc-stratified pair-level pilot. Re-extracts the 24-partition sample directly from velocity_data4 (rather than state_vectors_data4), exposing the DF17 BDS 0,9 source bit as the baro column (TRUE = barometric, FALSE = geometric/GNSS). Pairs are classified by the source bits at both endpoints into strict-barometric / strict-geometric / mixed-source strata, then enrichment and its within-cluster bootstrap CI are computed per stratum. • phase3_9_npair_regression.py (NEW, V38) — total_pairs-controlled cluster-robust logistic regression. Fits two models on the Dataset 3A low-latency sample: Model A (baseline flag_dv_rate ~ flag_dvr_rate + flag_dh_rate) and Model B (Model A plus log(total_pairs) covariate and flag_dvr_rate:log(total_pairs) interaction). Also produces a quintile-stratified pairwise enrichment table. Uses the canonical smf.logit(...).fit(cov_type='cluster', cov_kwds={'groups': icao24}, method='lbfgs') pattern shared with stage2_4_*.py and phase3_2_*.py. • phase3_engine_breakdown_audit.py — Strict-Fixed-Wing engine-type breakdown audit. • rename_dq_tier_columns.py — Utility for harmonising Stage 2.1 column naming across pipeline versions. 3. extractions.zip (Parquet shards) • Dataset_3A_aircraft_hour/ — Per-month Parquet shards of the 117,542,731 aircraft-hour aggregates, with eight integer counters per cluster {n, n_DV, n_DVR, n_DH, n_DV,DVR, n_DV,DH, n_DVR,DH, n_DV,DVR,DH}, median latitude/longitude, latency tier, fraction Δt = 1 s, and alternate-threshold columns for the DVR sweep. • Dataset_3B_300s_windows/ — Local two-month export of the 300-second tumbling-window aggregates. The canonical 12-month set used by the manuscript is in Windows_monthly_aggregates/. • Integrity_NACv_NIC_receiver/ — Per-month NACv/NIC/receiver-redundancy stratification source data. • pphysical_monthly_aggregates/ — Stage 2.1 output, identical schema to Dataset 3A. • Windows_monthly_aggregates/ — Stage 2.2 output, 12 monthly shards aggregating the 791,862,868 windows. • Spring_2024_re_extraction/ — Calendar-aligned 50-day Spring 2024 aircraft-hour aggregates (15,733,909 rows over 192,747 airframes). • Seasonal_checkpoint_extracts/ — Four one-week seasonal-checkpoint extracts (Winter, Spring, Summer, Autumn 2025) plus the Stage 2.4 / Phase 3.5 seasonal-windows extraction. • Aircraft_metadata_registry/enriched_aircraft_metadata.parquet — Stage 1 output: enriched aircraft registry cross-referenced across the OpenSky aircraft-database snapshots, FAA, Transport Canada, and Rikgale. 4. source_registries.zip (Source Registries) • aircraft_db/FAA/ — U.S. FAA Aircraft Registry (MASTER.txt, ACFTREF.txt). Public-domain US-government work (17 USC §105). • aircraft_db/ICAO/ — Rikgale-compiled ICAO Type Designators (rikgale_icao_types.csv). • aircraft_db/OpenSky/ — Eight monthly snapshots (May 2024 through February 2025) of the OpenSky aircraft database used to enrich tail-number provenance across the analysis window. • aircraft_db/Transport_Canada/ — Transport Canada Civil Aircraft Register (carscurr.txt). Government of Canada Open Government Licence. 5. RESULTS/ (Analytical Outputs, inside replication_kit.zip) Summaries (summaries/*.json): Per-phase machine-readable summaries containing population counts, headline enrichments, violation ratios with bootstrap confidence intervals, and per-stratum statistics. Includes phase3_8_vrsrc_pilot_summary.json and phase3_9_npair_regression_summary.json for the V38 additions. Manuscript Tables (manuscript_tables/*.csv): Direct mapping to the \label{tab:...} tags in the manuscript: CSV file Manuscript reference phase3_2_dq_tier_table.csv data-quality tier decomposition phase3_2_seasonal_table.csv seasonal robustness phase3_2_geographic_table.csv geographic robustness phase3_2_threshold_sweep.csv DVR threshold sweep phase3_2_strict_fw_regression.csv strict-Fixed-Wing regression phase3_2_helicopter_saturation_grid.csv rotary-wing saturation grid phase3_3_nacv_tier_table.csv, phase3_3_nic_tier_table.csv, phase3_3_receiver_tier_table.csv, phase3_3_joint_strict_table.csv, phase3_3_triple_intersection_table.csv NACv/NIC/receiver-redundancy stratification phase3_3_nac_strata.csv pair-level NACv pilot phase3_5_M1_observed_vs_null.csv permutation null observed-vs-null phase3_5_M2_threshold_sensitivity_table.csv symmetric-threshold sensitivity phase3_4b_d8b_spring2024_full_window.csv Spring 2024 year-over-year comparison phase3_engine_breakdown_strict.csv strict-Fixed-Wing engine breakdown phase3_8_vrsrc_strata.csv (NEW, V38) Table tab:pair_vrsrc — VrSrc-stratified pair-level enrichment (strict-barometric / strict-geometric / mixed-source) phase3_9_npair_regression_coefficients.csv (NEW, V38) Table tab:npair_regression — Model A baseline and Model B with log(total_pairs) + interaction, cluster-robust ORs and CIs phase3_9_npair_quintile_strata.csv (NEW, V38) Table tab:npair_quintile — pairwise enrichment stratified by total_pairs quintile (5 rows) Spring 2025 Diagnostics (spring2025_diagnostics/*.csv): Daily, weekly, hourly, geographic, altitude, airframe, and storm-events stratification of the 50-day elevation episode. Space-Weather Cross-Reference (space_weather/*.csv): Daily joined P(DV) and GFZ Potsdam indices, Pearson and Spearman correlations, storm-day coincidence summaries, and the top-ten extreme P(DV) days. Permutation Null (permutation_null/*.csv): Full 1,000-iteration null distribution of enrichment, V_A, V_I, V_B. Figures (figures/): File Manuscript reference Figure_1_permutation_null.pdf Figure 1 — permutation null distribution for the enrichment statistic Figure_2_saturation_grid_2panel.pdf Figure 2 — rotary-wing saturation grid (empirical vs Model 3B predicted) Figure_3_enrichment_forest.pdf Figure 3 — forest plot of pairwise enrichment across all stratifications Figure_4_spring_2025_timeline.pdf Figure 4 — daily P(DV), P(DVR), P(DH) for calendar 2025 with the 50-day elevation window and the daily sunspot number overlay Accompanying *_data.csv files contain the X-Y coordinate data plotted in each figure. Notes and Exclusions Manuscript LaTeX source is not included in this deposit. It is distributed separately via the journal upon acceptance. Pipeline execution logs are not included. They contain no information required for reproducibility; re-running the pipeline against the parquet shards regenerates all analytical outputs deterministically. icao_firs_global.geojson (FIR boundary polygons) is licensed and therefore not redistributed here. The geographic stratification is reproducible from the median latitude/longitude column retained in each Dataset 3A parquet shard, using the continental bounding boxes specified in config.py. Phase 3.8 and Phase 3.9 re-runs. Phase 3.8 (phase3_8_vrsrc_pilot.py) requires live access to the OpenSky Trino interface because it re-extracts at the raw-message level from the velocity_data4 table; supply OpenSky credentials via the standard config.py mechanism. Phase 3.9 (phase3_9_npair_regression.py) reads directly from the pphysical_*.parquet shards in extractions.zip and requires no network access; fit runtime is approximately 30 minutes for both models on a modern workstation. How to Cite If you use this dataset, pipeline, or any of its analytical outputs, please cite the accompanying manuscript: Pik, E. (2026). Multi-Dimensional Coupling of Kinematic Anomalies in ADS-B Surveillance Telemetry. Aerospace Science and Technology (under review). And the dataset itself: Pik, E. (2026). Replication dataset for "Multi-Dimensional Coupling of Kinematic Anomalies in ADS-B Surveillance Telemetry" (Version v1) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.20387308 Dataset DOI: 10.5281/zenodo.20387308 Acknowledgements The underlying state-vector telemetry is provided by the OpenSky Network (https://opensky-network.org) under their published terms of service. The NOAA Storm Events Database is publicly distributed by the U.S. National Centers for Environmental Information. The GFZ Potsdam daily geomagnetic and solar indices archive (Kp, ap, Ap, sunspot number, F10.7) is publicly distributed by GFZ Helmholtz Centre Potsdam. The FAA Aircraft Registry is a public-domain U.S.-government work; the Transport Canada Civil Aircraft Register is published under the Government of Canada Open Government Licence.