OpenTelemetry AIOps Benchmark: ML-based Anomaly Detection across Traces, Metrics, and Logs

Benchmark for an empirical study of machine learning anomaly detection across three OpenTelemetry signals (distributed traces, infrastructure metrics, and application logs), collected from microservice testbeds instrumented with OpenTelemetry on AWS EKS and subjected to a controlled fault-injection campaign (Chaos Mesh). Eight semi-supervised models are evaluated alongside a supervised Random Forest reference, using five-fold cross-validation by repetition with 2,550 Optuna hyperparameter-optimization trials. Models. Isolation Forest, One-Class SVM, LSTM Autoencoder, Transformer Autoencoder, 1D-CNN Autoencoder, LSTM-VAE, Deep SVDD, and DAGMM, with a supervised Random Forest baseline. Key findings. Under the engineered-feature representation, detection splits primarily by scoring mechanism rather than by signal type: the density-based deep models (DAGMM, Deep SVDD) discriminate on every signal, whereas the reconstruction-based autoencoders collapse to predict-all-anomaly on most signals. DAGMM is the strongest semi-supervised model on every signal under that representation (metrics F1 = 0.906, AUC-ROC = 0.960) and, under a matched evaluation condition, equals or exceeds the supervised Random Forest F1 without using anomaly labels in training. Because the reconstruction collapse yields high F1 at near-random AUC-ROC, AUC-ROC is the recommended ranking metric. The log ranking is representation-dependent: a Drain/TF-IDF/PCA representation lifts the Transformer Autoencoder above DAGMM on logs. Contents (single archive otel-aiops-benchmark.zip). data/ holds processed feature files (Parquet), per-model/signal/fold results, raw anomaly scores, chaos-injection configurations, and the additional analysis result tables (inference cost, log representation, threshold robustness, metric reconciliation, prevalence-normalized comparison, window-size sensitivity, Friedman ranking, log feature importance). code/ holds the feature-engineering and model-training pipeline, the analysis scripts that compute the reported metrics, and the figure-generation scripts. A README documents the layout. Cloud account identifiers and execution-role ARNs in the pipeline scripts are read from environment variables with neutral placeholders. v3.1.0 adds the cross-dataset external-validity code and derived results (paper5_external_validity_code_v3.1.0.zip): RCAEval RE2 and Alibaba 2021 feature extractors, the eight-model replication runner, and the combined-signal density-model fusion script, plus the per-run and summary result tables. Third-party raw telemetry is not redeposited; obtain RCAEval RE2 from DOI 10.5281/zenodo.14590730 and the Alibaba 2021 microservices traces from github.com/alibaba/clusterdata.