GWDG GPU Node Telemetry Dataset for Observability-Aware Early Warning of GPU Detachment Failures (2025-2026)

This dataset provides a large-scale, structured collection of sanitized time-series telemetry from production GPU-equipped HPC nodes at the GWDG high-performance computing infrastructure. It is aligned with operator-curated GPU failure incidents observed between January 2025 and February 2026, supporting reproducible research on observability-aware early warning of GPU instability. The dataset, titled “GWDG GPU Node Telemetry Dataset for Observability-Aware Early Warning of GPU Detachment Failures (2025–2026)” (DOI: 10.5281/zenodo.19052367), has been substantially expanded in v2 and contains: 50 incident telemetry windows 50 per-incident telemetry files with corresponding metadata 19 unique GPU nodes 69 incidents in the complete operator incident catalog 10-second telemetry sampling, compared with 10-minute sampling in v1 Approximately 9,361 timestamps per incident window, spanning roughly 25-27 hours A continuous scheduler-state dataset covering the full observation period Version 2 is a higher-resolution re-extraction of the underlying Prometheus archives. It retains all metric families present in the export, adds IPMI/BMC and scheduler-state telemetry, introduces stable series identifiers for multi-series metrics, and anchors incidents using Slurm-detected state transitions rather than only operator-recorded machine-log dates. Data Structure and Observability Dimensions Each telemetry record captures multiple operational dimensions: Temporal dimension: timeUtc contains true UTC timestamps, while utcOffsetHours records the corresponding Europe/Berlin UTC offset Node and hardware context: node identity is encoded through the pseudonymized filename and metadata; GPU identifiers and UUIDs remain pseudonymized Metric definition: metric, value Series context: seriesId, resolvable through series_labels.tsv, distinguishes concurrent time series belonging to the same metric family Workload context: scheduler- and job-related signals link telemetry behavior to system activity Monitoring infrastructure: scrape indicators such as up expose monitoring-pipeline availability and degradation Hardware/software configuration: GPU and system telemetry provide device, model, driver, and hardware-state context where available Scheduler context: slurm_node_state_tidy.csv.xz provides node-state information across the full observation period Per-incident telemetry tables no longer contain a redundant node column. Each file represents one node, whose pseudonym is encoded directly in the filename and recorded in the corresponding metadata. The Prometheus instance column has also been removed because it acted as a second identifier for the same host. This structure enables multi-dimensional analysis of GPU behavior across hardware, workload, scheduler, and monitoring layers while avoiding redundant host identifiers. Observability-Aware Failure Modeling Unlike traditional datasets that focus solely on numeric anomalies, this dataset explicitly supports failure modes where observability itself degrades. These include: metric disappearance scrape instability and failures exporter unavailability monitoring-pipeline inconsistencies structural gaps in expected telemetry disappearance of entire telemetry planes Such conditions are particularly relevant for GPU detachment and PCIe/bus-related failures, where a GPU may become inaccessible without producing a clear numeric precursor. Version 2 preserves signals that were excluded from the original extraction, including scrape indicators, per-CPU statistics, filesystem and system metrics, IPMI/BMC telemetry, and other metric families that may reveal structural changes in system observability. Stable seriesId values also prevent distinct Prometheus series from collapsing into indistinguishable rows when their differentiating labels are not represented directly in the tidy telemetry schema. Incident Coverage The complete incident_events.csv catalog contains 69 operator-recorded incidents. Of these: 50 incidents have corresponding per-incident telemetry files 19 incidents are marked not-collected because telemetry was never gathered for them incidents span multiple GPU-related failure categories, including GPU malfunction, GPU disappearance, detachment, and PCIe/bus-related failures Incident timing in v2 is more precise than in the original release. The incident catalog includes: slurmDetectedTime, representing the scheduler-detected failure transition slurmPreviousHealthyTime, representing the preceding known healthy scheduler state Together, these timestamps bracket the onset of a failure and provide a stronger temporal anchor for early-warning experiments than the machine-log date used in v1. Four collected incidents contain no DCGM telemetry plane because the GPU exporter was not scraping the affected node during the corresponding window. This absence is preserved intentionally because it represents an observability-collapse signal, rather than a processing defect. The five quarterly cluster-wide “GPU when good” tables included in v1 are no longer part of v2. They are superseded by the continuous scheduler-state dataset, which provides per-node state information across the complete observation period and enables more flexible construction of healthy or reference intervals. Observability Sources The telemetry is derived primarily from Prometheus monitoring archives and integrates multiple observability planes: GPU telemetry via NVIDIA DCGM Node and OS telemetry via Prometheus node exporter Chassis and BMC telemetry via IPMI, including fan speed, temperature, voltage, power, and power-state measurements Monitoring-pipeline telemetry, including scrape indicators such as up Scheduler-state signals derived from Slurm Each per-incident telemetry file contains a high-resolution window sampled at 10-second cadence, preserving temporal context around the corresponding failure event. The separate slurm_node_state_tidy.csv.xz table spans nodes across the broader observation period, allowing scheduler behavior to be analyzed independently of individual incident windows. Timestamp Semantics A timestamp issue present in v1 has been corrected in v2. In v1, the column named timeUtc actually contained Europe/Berlin local wall-clock time, even though the Prometheus query itself used the correct UTC interval. In v2: timeUtc contains true UTC utcOffsetHours stores the local offset in effect at that timestamp: +1 during CET +2 during CEST When comparing v1 and v2 telemetry, v1 timestamps should therefore be aligned against: timeUtc + utcOffsetHours rather than directly against the v2 timeUtc column. Data Sanitization and Privacy All published node and GPU identifiers are pseudonymized using deterministic salted hashing. The underlying pseudonymization salt and digest are unchanged from v1, so identifiers in both versions refer to the same entities. For filename safety and unambiguous parsing, v2 removes the underscore immediately following identifier prefixes. For example: node_2b08d249c0ba in v1 becomes node2b08d249c0ba in v2. The dataset is sanitized to avoid disclosure of user data, credentials, network topology information, application-level traces, and other sensitive infrastructure identifiers. Data Distribution and Integrity Per-incident telemetry is distributed as xz-compressed tidy CSV files using the naming convention: <node>_<date>_<category>_tidy.csv.xz Corresponding metadata files use: <node>_<date>_<category>_meta.json Additional dataset-level files include: slurm_node_state_tidy.csv.xz - scheduler state across nodes and the full observation period series_labels.tsv - mapping from seriesId to distinguishing Prometheus labels incident_events.csv - complete 69-incident operator catalog manifest.csv - per-file coverage, timing, category, cadence, and file statistics SHA256SUMS.txt - checksums for published files CITATION.cff CITATION.bib Telemetry tables use xz compression rather than the bzip2 compression used in v1. Standard tools such as pandas can read .csv.xz files directly. Research Applications With substantially higher temporal resolution, broader incident coverage, additional observability planes, and explicit representation of monitoring degradation, this dataset provides a foundation for: observability-aware anomaly detection early warning systems for GPU detachment failures predictive maintenance in GPU-enabled HPC systems detection of structural telemetry collapse multimodal failure detection across GPU, host, BMC, monitoring, and scheduler signals benchmarking reliability and monitoring frameworks studying missingness and metric disappearance as first-class failure signals evaluating failure-detection methods under partial or degraded observability The combination of numerical telemetry and structural observability signals makes the dataset particularly suitable for studying failures in which the disappearance of expected measurements may be more informative than anomalous measurement values. Related Work This dataset accompanies the study: Bidollahkhani, M., Nordsiek, F., & Kunkel, J. M. (2026).“When GPUs Fail Quietly: Observability-Aware Early Warning Beyond Numeric Telemetry.”arXiv: https://arxiv.org/abs/2603.28781