RCAEval: A Benchmark for Root Cause Analysis of Microservice Systems

RCAEval benchmark includes nine datasets organized into three benchmark suites (RE1, RE2, RE3), each covering three microservice systems (Online Boutique, Sock Shop, Train Ticket). Together, these datasets feature 735 failure cases with 11 fault types. Each failure case includes annotated root cause service and root cause indicator (e.g., specific metric or log indicating the root cause).RE1 Datasets (375 cases). The RE1 datasets, introduced in our prior work on metric-based RCA, contain 375 failure cases collected from three microservice systems (125 cases per system). Each system includes five fault types across five services, with five repetitions per fault-service pair. RE1 exclusively contains metrics data, supporting the development of metric-based RCA methods. The fault types include CPU, MEM, DISK, DELAY, and LOSS. The number of metrics ranges from 49 to 238 depending on the system size.- RE1-OB (Online Boutique): 125 cases, 5 services (adservice, cartservice, checkoutservice, currencyservice, productcatalogservice)- RE1-SS (Sock Shop): 125 cases, 5 services (carts, catalogue, orders, payment, user)- RE1-TT (Train Ticket): 125 cases, 5 services (ts-auth-service, ts-order-service, ts-route-service, ts-train-service, ts-travel-service)RE2 Datasets (270 cases). The RE2 datasets, newly collected for this study, support the development of multi-source RCA methods. They include 270 failure cases collected from three microservice systems (90 cases per system), combining six fault types across five services, with three repetitions per fault-service pair. RE2 provides multi-source telemetry data including metrics, logs, and traces. The fault types include those in RE1 plus an additional SOCKET fault.- RE2-OB (Online Boutique): 90 cases, 69-77 metrics, with logs and traces- RE2-SS (Sock Shop): 90 cases, 74-82 metrics, with logs and traces- RE2-TT (Train Ticket): 90 cases, 340-376 metrics, with logs and tracesRE3 Datasets (90 cases). The RE3 datasets, also newly collected, focus on supporting multi-source RCA methods with the ability to diagnose code-level faults. They contain 90 failure cases (30 per system) involving code-level faults (F1, F2, F3, F4, F5). Like RE2, RE3 includes multi-source telemetry data (metrics, logs, and traces). This dataset emphasizes diagnosing code-level faults through telemetry data, e.g., leveraging stack traces in logs or response codes in traces to pinpoint root causes.- RE3-OB (Online Boutique): 30 cases, 68-101 metrics, with logs and traces- RE3-SS (Sock Shop): 30 cases, 80-107 metrics, with logs and traces- RE3-TT (Train Ticket): 30 cases, 294-322 metrics, with logs and traces---File Structure:Each dataset directory follows the naming convention: {benchmark}_{service}_{fault}_{instance}- metrics.json: Time-series metrics data- inject_time.txt: Fault injection timestamp (Unix timestamp)- logs.csv: Log data (RE2 and RE3 only)- traces.csv: Trace data (RE2 and RE3 only)The benchmark evaluation framework supporting these datasets is available at: https://github.com/phamquiluan/RCAEval