Degradation of ADS-B Surveillance Integrity: Temporal Clustering and Anomaly Persistence

Data License: Creative Commons Attribution 4.0 International (CC BY 4.0) Code License: MIT Overview Welcome to the replication dataset for the study "Degradation of ADS-B Surveillance Continuity: Temporal Clustering and Anomaly Persistence". This repository contains the complete open-science codebase, raw extraction shards, pre-computed survival matrices, and statistical artifacts required to fully reproduce the longitudinal analysis of 2025 global ADS-B tracking stability. Extracting from a gross volume of 246.7 billion OpenSky Network ADS-B messages logged across the 2025 calendar year, this study analyzes 229.0 billion filtered, airborne state vectors. The customized, out-of-core computational pipeline executes scrambled-temporal permutation testing ($\mathcal{W}=\{3,5,7,10,15\}$), time-to-recurrence Kaplan-Meier survival modeling, and exposure-adjusted Cox proportional hazards regression to quantify longitudinal surveillance continuity across 233,859 unique aircraft trajectories. Quick Start & Reproducibility To replicate the temporal clustering and survival modeling pipeline locally: Install dependencies: pip install -r requirements.txt Execute Stage 1: Run CODE/stage1_trino_extraction.py on the OpenSky Trino cluster to extract sequential data (Note: Requires OpenSky credentials). Execute Stage 2: Run CODE/stage2_data_generation.py to execute the Numba JIT-compiled permutation tests and build the survival cohorts. Execute Stage 3: Run CODE/stage3_figure_generation.py to ingest the Stage 2 .parquet files and rapidly re-render all manuscript PDFs and CSV tables. Directory Structure . ├── README.md ├── requirements.txt ├── CODE/ │ ├── config.py │ ├── extract_manuscript_data.py │ ├── stage1_trino_extraction.py │ ├── stage2_data_generation.py │ └── stage3_figure_generation.py ├── DATA/ │ ├── H2_enhanced_results.parquet │ └── extractions/ │ ├── Dataset_2A_aggregated_2025XXXX.PARQUET.zip │ ├── Dataset_2B_daily_spans_2025XXXX.PARQUET.zip │ ├── Dataset_2B_ledger_2025XXXX.PARQUET.zip │ └── Q1_diagnostic_stats_2025XXXX.CSV.zip ├── LOGS/ │ ├── stage1_trino_extraction.log │ ├── stage2_data_generation.log │ └── stage3_figure_generation.log └── RESULTS/ ├── data_permutation_validation.csv ├── df_gaps_raw.parquet ├── df_ledger.parquet ├── fig_censoring_sensitivity_data.csv ├── fig_continental_sensitivity_data.csv ├── fig_exposure_stratified_survival_data.csv ├── fig_stratified_survival_data.csv ├── gap_stats.parquet ├── manuscript_fill_values.json ├── Master_Survival_Cohort.parquet ├── model_cox_coefficients.csv ├── model_schoenfeld_residuals.txt ├── tab_block_sensitivity.csv ├── tab_censoring_demographics.csv ├── tab_per_aircraft_clustering.csv └── figures/ ├── Figure_1_pipeline_architecture.pdf ├── Figure_2_stratified_recurrence.pdf ├── Figure_3_cox_exposure_stratified.pdf ├── Figure_S1_censoring_sensitivity.pdf ├── Figure_S2_dropout_sensitivity.pdf ├── Figure_S3_penalizer_sensitivity.pdf └── Figure_S4_continental_sensitivity.pdf File Manifest & Data Dictionary 1. Root Files README.md: This documentation file. requirements.txt: Standardized list of Python dependencies (e.g., numba, lifelines, pyarrow) to ensure environment reproducibility. 2. CODE/ (The Computational Pipeline) config.py: The central configuration module. Contains the exact kinematic anomaly thresholds (e.g., speed $> 6670.3$ m/s), memory-guard limits, and temporal window size parameters used universally across all pipeline stages. extract_manuscript_data.py: A parsing utility that reads the logs, CSVs, and Parquet files to dynamically extract and format the macro-statistics (like total flight hours and fleet sizes) needed for the manuscript text. stage1_trino_extraction.py: The distributed database extraction script. Executes native SQL window functions on the OpenSky Trino cluster to download and pair daily trajectory trajectories. stage2_data_generation.py: The core data science engine. Runs the out-of-core scrambled-temporal permutation tests (via Numba JIT compilation) and aggregates raw trajectories into discrete monthly survival datasets. stage3_figure_generation.py: The visualization engine. Ingests Stage 2 data to compute the Cox Proportional Hazards models, log-rank tests, and renders the publication-ready PDF figures and CSV tables. 3. DATA/ (Inputs and Intermediates) H2_enhanced_results.parquet: The pre-computed master results matrix from the Stage 2 permutation tests, containing Z-statistics, O/E ratios, and FDR-corrected p-values for all evaluated window scales. extractions/Dataset_2A_aggregated_*.PARQUET.zip: Raw binary anomaly vectors used for independent multi-scale testing. extractions/Dataset_2B_daily_spans_*.PARQUET.zip: Daily observation boundaries used to track network dropouts. extractions/Dataset_2B_ledger_*.PARQUET.zip: The severity-classified anomaly ledger. extractions/Q1_diagnostic_stats_*.CSV.zip: Daily data quality reports (e.g., duplicate message rates). 4. LOGS/ stage1_trino_extraction.log: Extraction timing and data volume logs. stage2_data_generation.log: Permutation tracking and cohort construction logs. stage3_figure_generation.log: Modeling outputs, including the in-sample Cox Model C-Index. 5. RESULTS/ (Final Datasets, Tables, and Models) Master_Survival_Cohort.parquet: The core discrete life-table dataset. It anchors $T_0$ to individual observation inception to prevent immortal time bias and provides the basis for all Kaplan-Meier and Cox models. df_ledger.parquet: Reconciled monthly ledger mapping raw anomalies to their specific physical Severity Classes. df_gaps_raw.parquet: Consolidated raw daily observation spans. gap_stats.parquet: Processed inter-observation gaps used to enforce 30-day network dropout right-censoring. data_permutation_validation.csv: Results of the empirical validation subset confirming high classification concordance between $N=200$ and $N=10,000$ permutations. model_cox_coefficients.csv: Complete Cox Proportional Hazards regression outputs ($\beta$ coefficients, hazard ratios, $p$-values). model_schoenfeld_residuals.txt: Outputs from the scaled Schoenfeld residuals test evaluating the proportional hazards assumption. manuscript_fill_values.json: An auto-generated JSON key-value store containing all dynamic scalars referenced directly in the manuscript body. 6. RESULTS/ (Manuscript Tables & Curve Coordinates) tab_per_aircraft_clustering.csv: Per-aircraft temporal clustering statistics across all window scales. tab_block_sensitivity.csv: Comparison of significance rates evaluating the uniform-null versus the stricter block-preserving null model. tab_censoring_demographics.csv: Demographic profiles of the resulting survival cohorts by censoring status. fig_stratified_survival_data.csv: Coordinate data for Figure 2 (Severity Class stratification). fig_exposure_stratified_survival_data.csv: Coordinate data for Figure 3 (Flight hours tertile stratification). fig_censoring_sensitivity_data.csv: Coordinate data for Figure S1 (Complete cases vs baseline). fig_continental_sensitivity_data.csv: Coordinate data for Figure S4 (Continental-only subset). 7. RESULTS/figures/ (Publication Graphics) Figure_1_pipeline_architecture.pdf: Methodological flowchart. Figure_2_stratified_recurrence.pdf: Time-to-first anomaly survival functions by physical Error Mode. Figure_3_cox_exposure_stratified.pdf: Exposure-adjusted survival hazard stratifications. Figure_S1_censoring_sensitivity.pdf: Sensitivity impact of network dropouts vs administrative censoring. Figure_S2_dropout_sensitivity.pdf: Stability check across 15, 30, 45, and 60-day network dropout thresholds. Figure_S3_penalizer_sensitivity.pdf: Model coefficient stability versus L2 Ridge penalizer variance. Figure_S4_continental_sensitivity.pdf: Sensitivity control isolating oceanic line-of-sight dropouts. How to Cite If you utilize this dataset, pipeline architecture, or empirical baseline in your research, please cite the accompanying manuscript: Pik, E. (2026). Degradation of ADS-B Surveillance Continuity: Temporal Clustering and Anomaly Persistence. Aerospace Science and Technology (Under Review). Dataset DOI: 10.5281/zenodo.19698275数据许可证：知识共享署名4.0国际许可协议（Creative Commons Attribution 4.0 International，CC BY 4.0） 代码许可证：MIT ## 概述 欢迎使用研究《ADS-B监视连续性退化：时间聚类与异常持续性》的复现数据集。本仓库包含完整的开放科学代码库、原始提取分片、预计算生存矩阵以及统计成果集，可完全复现2025年全球自动相关监视广播（Automatic Dependent Surveillance-Broadcast，ADS-B）跟踪稳定性的纵向分析。 本研究从2025日历年期间记录的2467亿条开放天空网络（OpenSky Network）ADS-B原始消息中提取数据，经筛选后分析了2290亿条机载状态向量。定制化的外核计算流水线执行打乱时间置换检验（$mathcal{W}={3,5,7,10,15}$）、复发时间Kaplan-Meier生存建模以及暴露校正Cox比例风险回归，以量化233859条独特航空器轨迹的纵向监视连续性。 ## 快速开始与可复现性 若需在本地复现时间聚类与生存建模流水线： 1. 安装依赖：`pip install -r requirements.txt` 2. 执行阶段1：在OpenSky Trino集群上运行`CODE/stage1_trino_extraction.py`以提取序列数据（注：需OpenSky凭证）。 3. 执行阶段2：运行`CODE/stage2_data_generation.py`以执行Numba JIT编译的置换检验并构建生存队列。 4. 执行阶段3：运行`CODE/stage3_figure_generation.py`以读取阶段2的Parquet文件，快速重新生成所有论文手稿PDF与CSV表格。 ## 目录结构 . ├── README.md ├── requirements.txt ├── CODE/ │ ├── config.py │ ├── extract_manuscript_data.py │ ├── stage1_trino_extraction.py │ ├── stage2_data_generation.py │ └── stage3_figure_generation.py ├── DATA/ │ ├── H2_enhanced_results.parquet │ └── extractions/ │ ├── Dataset_2A_aggregated_2025XXXX.PARQUET.zip │ ├── Dataset_2B_daily_spans_2025XXXX.PARQUET.zip │ ├── Dataset_2B_ledger_2025XXXX.PARQUET.zip │ └── Q1_diagnostic_stats_2025XXXX.CSV.zip ├── LOGS/ │ ├── stage1_trino_extraction.log │ ├── stage2_data_generation.log │ └── stage3_figure_generation.log └── RESULTS/ ├── data_permutation_validation.csv ├── df_gaps_raw.parquet ├── df_ledger.parquet ├── fig_censoring_sensitivity_data.csv ├── fig_continental_sensitivity_data.csv ├── fig_exposure_stratified_survival_data.csv ├── fig_stratified_survival_data.csv ├── gap_stats.parquet ├── manuscript_fill_values.json ├── Master_Survival_Cohort.parquet ├── model_cox_coefficients.csv ├── model_schoenfeld_residuals.txt ├── tab_block_sensitivity.csv ├── tab_censoring_demographics.csv ├── tab_per_aircraft_clustering.csv └── figures/ ├── Figure_1_pipeline_architecture.pdf ├── Figure_2_stratified_recurrence.pdf ├── Figure_3_cox_exposure_stratified.pdf ├── Figure_S1_censoring_sensitivity.pdf ├── Figure_S2_dropout_sensitivity.pdf ├── Figure_S3_penalizer_sensitivity.pdf └── Figure_S4_continental_sensitivity.pdf ## 文件清单与数据字典 ### 1. 根目录文件 - README.md：本文档文件。 - requirements.txt：标准化Python依赖列表（例如numba、lifelines、pyarrow），确保环境可复现。 ### 2. CODE/（计算流水线） - config.py：核心配置模块，包含全流水线各阶段统一使用的精确运动学异常阈值（例如速度>6670.3 m/s）、内存保护限制与时间窗口大小参数。 - extract_manuscript_data.py：解析工具，可读取日志、CSV与Parquet文件，动态提取并格式化手稿文本所需的宏观统计量（如总飞行时长与机队规模）。 - stage1_trino_extraction.py：分布式数据库提取脚本，在OpenSky Trino集群上执行原生SQL窗口函数，下载并配对每日航空器轨迹。 - stage2_data_generation.py：核心数据科学引擎，运行外核打乱时间置换检验（通过Numba JIT编译），并将原始轨迹聚合为离散月度生存数据集。 - stage3_figure_generation.py：可视化引擎，读取阶段2数据以计算Cox比例风险模型、对数秩检验，并生成可直接用于出版的PDF图表与CSV表格。 ### 3. DATA/（输入与中间产物） - H2_enhanced_results.parquet：阶段2置换检验预计算的主结果矩阵，包含所有评估窗口尺度的Z统计量、观测/预期比值比与错误发现率校正p值。 - extractions/Dataset_2A_aggregated_*.PARQUET.zip：用于独立多尺度测试的原始二进制异常向量。 - extractions/Dataset_2B_daily_spans_*.PARQUET.zip：用于追踪网络中断的每日观测边界。 - extractions/Dataset_2B_ledger_*.PARQUET.zip：按严重程度分类的异常记录册。 - extractions/Q1_diagnostic_stats_*.CSV.zip：每日数据质量报告（例如重复消息率）。 ### 4. LOGS/（日志文件） - stage1_trino_extraction.log：提取耗时与数据量日志。 - stage2_data_generation.log：置换检验追踪与队列构建日志。 - stage3_figure_generation.log：建模输出日志，包括样本内Cox模型C指数。 ### 5. RESULTS/（最终数据集、表格与模型） - Master_Survival_Cohort.parquet：核心离散寿命表数据集，将$T_0$锚定至个体观测起始时刻以避免不朽时间偏差，为所有Kaplan-Meier与Cox模型提供基础。 - df_ledger.parquet：经核对的月度记录册，将原始异常映射至其对应的物理严重程度类别。 - df_gaps_raw.parquet：整合的原始每日观测跨度。 - gap_stats.parquet：用于实施30天网络中断右删失的处理后观测间隙统计量。 - data_permutation_validation.csv：经验验证子集的结果，确认$N=200$与$N=10000$次置换间具有高度分类一致性。 - model_cox_coefficients.csv：完整的Cox比例风险回归输出（$eta$系数、风险比、p值）。 - model_schoenfeld_residuals.txt：缩放Schoenfeld残差检验的输出，用于评估比例风险假设的合理性。 - manuscript_fill_values.json：自动生成的JSON键值存储库，包含手稿正文中直接引用的所有动态标量值。 - tab_per_aircraft_clustering.csv：所有窗口尺度下的单航空器时间聚类统计量。 - tab_block_sensitivity.csv：对比均匀零假设与更严格的块保留零模型的显著性率。 - tab_censoring_demographics.csv：按删失状态划分的生存队列人口统计特征。 - fig_stratified_survival_data.csv：图2（严重程度类别分层）的坐标数据。 - fig_exposure_stratified_survival_data.csv：图3（飞行时长三分位数分层）的坐标数据。 - fig_censoring_sensitivity_data.csv：补充图S1（完整案例与基线）的坐标数据。 - fig_continental_sensitivity_data.csv：补充图S4（仅大陆子集）的坐标数据。 ### 6. RESULTS/figures/（出版级图表） - Figure_1_pipeline_architecture.pdf：方法学流程图。 - Figure_2_stratified_recurrence.pdf：按物理错误模式划分的首次异常复发时间生存函数。 - Figure_3_cox_exposure_stratified.pdf：暴露校正的生存风险分层。 - Figure_S1_censoring_sensitivity.pdf：网络中断与管理删失的敏感性影响。 - Figure_S2_dropout_sensitivity.pdf：15、30、45与60天网络中断阈值下的稳定性检验。 - Figure_S3_penalizer_sensitivity.pdf：模型系数稳定性与L2岭惩罚方差的关系。 - Figure_S4_continental_sensitivity.pdf：隔离视距范围内海洋区域中断的敏感性控制分析。 ## 引用方式 若您在研究中使用本数据集、流水线架构或经验基线，请引用随附的手稿： Pik, E. (2026). Degradation of ADS-B Surveillance Continuity: Temporal Clustering and Anomaly Persistence. Aerospace Science and Technology (Under Review). 数据集DOI：10.5281/zenodo.19698275