Streaming Pipeline Anomaly Detection Benchmark: Kafka and Flink Telemetry Dataset

Benchmark dataset for "ML-based anomaly detection for data streaming pipeline reliability: an empirical benchmark on Apache Kafka and Flink." This dataset accompanies an empirical study evaluating six semi-supervised ML models, a supervised Random Forest baseline, and a statistical threshold baseline for detecting anomalies in Apache Kafka and Apache Flink streaming pipelines deployed on Amazon EKS. Contents:- Feature files (Parquet) for two workload levels: 5,000 events/s (primary) and 500 events/s (cross-workload validation), with window sizes of 15s, 30s, 60s, and 120s- Model results (CSV): per-fold F1, AUC-ROC, precision, recall, detection latency, confusion matrices, unsupervised threshold evaluation, RF feature importance, and Optuna hyperparameter logs- Cross-workload comparison: Wilcoxon signed-rank tests with Holm-Bonferroni correction across 8 models at both throughput levels- Post-hoc analyses: prevalence sensitivity, cooldown exclusion, score fusion, pairwise statistical tests- Campaign manifests (JSON): fault injection logs for all 80 runs (8 faults x 10 repetitions) at each workload level- Infrastructure code: Terraform (EKS), Helm values, Kubernetes manifests, campaign orchestrator- Streaming application: Python load generator, 3 Java/Gradle Flink jobs (enrichment, aggregation, fraud detection), Avro schemas- Analysis scripts: preprocessing, training, merging, statistical tests, cross-workload comparison, figure and table generation- Manuscript figures (PDF): all 9 figures as used in the paper Experimental design:- 10-fold leave-one-repetition-out cross-validation with a 4-role split (train / hpo_val / cal / test) eliminating the validation triple-use problem- 8 streaming-specific fault types injected via Chaos Mesh: broker crash, network partition, disk I/O saturation, consumer lag spike, rebalancing storm, poison pill, checkpoint failure, OOM exhaustion- 80 controlled fault injection runs per workload level (8 faults x 10 repetitions)- Models: Isolation Forest, One-Class SVM, LSTM Autoencoder, Transformer Autoencoder, 1D-CNN Autoencoder, LSTM-VAE, Random Forest (supervised), statistical threshold baseline- Metrics collected via Prometheus (10s scrape) from Kafka JMX, Flink metrics reporter, and JVM exporters