OpenTelemetry AIOps Benchmark: ML-based Anomaly Detection across Traces, Metrics, and Logs

Benchmark for an empirical study of machine learning anomaly detection across three OpenTelemetry signals (distributed traces, infrastructure metrics, and application logs), collected from microservice testbeds instrumented with OpenTelemetry on AWS EKS and subjected to a controlled fault-injection campaign (Chaos Mesh). Eight semi-supervised models are evaluated alongside a supervised Random Forest reference, using five-fold cross-validation by repetition with 2,550 Optuna hyperparameter-optimization trials. Models. Isolation Forest, One-Class SVM, LSTM Autoencoder, Transformer Autoencoder, 1D-CNN Autoencoder, LSTM-VAE, Deep SVDD, and DAGMM, with a supervised Random Forest baseline. Key findings. Under the engineered-feature representation, detection splits primarily by scoring mechanism rather than by signal type: the density-based deep models (DAGMM, Deep SVDD) discriminate on every signal, whereas the reconstruction-based autoencoders collapse to predict-all-anomaly on most signals. DAGMM is the strongest semi-supervised model on every signal under that representation (metrics F1 = 0.906, AUC-ROC = 0.960) and, under a matched evaluation condition, equals or exceeds the supervised Random Forest F1 without using anomaly labels in training. Because the reconstruction collapse yields high F1 at near-random AUC-ROC, AUC-ROC is the recommended ranking metric. The log ranking is representation-dependent: a Drain/TF-IDF/PCA representation lifts the Transformer Autoencoder above DAGMM on logs. Contents (single archive otel-aiops-benchmark.zip). data/ holds processed feature files (Parquet), per-model/signal/fold results, raw anomaly scores, chaos-injection configurations, and the additional analysis result tables (inference cost, log representation, threshold robustness, metric reconciliation, prevalence-normalized comparison, window-size sensitivity, Friedman ranking, log feature importance). code/ holds the feature-engineering and model-training pipeline, the analysis scripts that compute the reported metrics, and the figure-generation scripts. A README documents the layout. Cloud account identifiers and execution-role ARNs in the pipeline scripts are read from environment variables with neutral placeholders.