A Deep Look Into the Temporal I/O Behavior of HPC Applications [Dataset]

A Deep Look Into the Temporal I/O Behavior of HPC Applications [Dataset]

This repository contains the code and dataset used in the **IPDPS 2025** paper titled *"A Deep Look Into the Temporal I/O Behavior of HPC Applications."* The repository is organized as follows:

The signal files contain the I/O traces used in this study.

The clusterwise-master contains the Python tools used to generate and analyze the traces.

The data tar file contains some data extracted from the traces used to conduct the studies presented in the paper. Besides the data, it also includes Python code for generating graphs and analyses.

The Signal Tar Files

The signal files consist of I/O traces collected from two HPC systems:

PlaFRIM

PlaFRIM is an experimental platform from the Inria Center at the University of Bordeaux. It has 192 nodes and a BeeGFS storage system with two OSSs, each with four OSTs, a default stripe count of 4, and a 100 Gbps network. Its peak I/O performance is approximately 12 GiB/s.  We collected data over a period of 26 months (from May 2022 to July 2024). The `beegfs-ctl` command was used to obtain bandwidth usage (grouped by user) every second.

SDumont

SDumont is located at the National Laboratory for Scientific Computing (LNCC) in Brazil. It consists of 36,472 cores distributed across 1,134 nodes and has a peak performance of 5.1 petaflops.  Its Lustre storage system is deployed on 10 OSSs, each with one OST, a default stripe count of 1, and a peak I/O performance of 30 GiB/s. Data collection was performed using Collectl, which gathered information from each compute node every 15 seconds, spanning 12 months (from January 2020 to December 2020).

Dataset Format

Plafrim and Sdumont files follow a CSV format with three columns: read, write, and both.

The read and write columns represent the I/O operations executed.  

The values in these columns indicate the bandwidth of the operation: 

PlaFRIM: MiB/s  

SDumont: KiB/s

The both column contains the sum of the bandwidth for read and write operations.

Each line in the file represents a 1-second measurement.

Each file in the dataset corresponds to a job, identified by a unique file ID

Blue Waters

In this paper, we also use Blue Waters traces. These traces can be acquired at https://bluewaters.ncsa.illinois.edu/data-sets

ClusterWise Project

ClusterWise is a Python project that contains all the code we wrote to generate and parse the traces. The project includes a README file with more information on how to use it. You can also check the `--help` option in the `clusterWise.py` file.

The Data Tar File

For the analyses conducted in the paper, we generated some intermediate files. For instance, the output of the FTIO tool when studying the periodicity of the I/O. These files can be found in the `data.tar.gz` file. Moreover, it also contains Jupyter notebooks and Python code used to analyze the data.

FTIO Artifacts Reproducibility

Several results presented in the paper were generated using the FTIO tool. To fully reproduce these artifacts, please follow the steps in the repository: FTIO Artifacts Repository.