A Deep Look Into the Temporal I/O Behavior of HPC Applications [Dataset]

A Deep Look Into the Temporal I/O Behavior of HPC Applications [Dataset] This repository contains the code and dataset used in the **IPDPS 2025** paper titled *"A Deep Look Into the Temporal I/O Behavior of HPC Applications."* The repository is organized as follows: The signal files contain the I/O traces used in this study. The clusterwise-master contains the Python tools used to generate and analyze the traces. The data tar file contains some data extracted from the traces used to conduct the studies presented in the paper. Besides the data, it also includes Python code for generating graphs and analyses. The Signal Tar Files The signal files consist of I/O traces collected from two HPC systems: PlaFRIM PlaFRIM is an experimental platform from the Inria Center at the University of Bordeaux. It has 192 nodes and a BeeGFS storage system with two OSSs, each with four OSTs, a default stripe count of 4, and a 100 Gbps network. Its peak I/O performance is approximately 12 GiB/s. We collected data over a period of 26 months (from May 2022 to July 2024). The `beegfs-ctl` command was used to obtain bandwidth usage (grouped by user) every second. SDumont SDumont is located at the National Laboratory for Scientific Computing (LNCC) in Brazil. It consists of 36,472 cores distributed across 1,134 nodes and has a peak performance of 5.1 petaflops. Its Lustre storage system is deployed on 10 OSSs, each with one OST, a default stripe count of 1, and a peak I/O performance of 30 GiB/s. Data collection was performed using Collectl, which gathered information from each compute node every 15 seconds, spanning 12 months (from January 2020 to December 2020). Dataset Format Plafrim and Sdumont files follow a CSV format with three columns: read, write, and both. The read and write columns represent the I/O operations executed. The values in these columns indicate the bandwidth of the operation: PlaFRIM: MiB/s SDumont: KiB/s The both column contains the sum of the bandwidth for read and write operations. Each line in the file represents a 1-second measurement. Each file in the dataset corresponds to a job, identified by a unique file ID Blue Waters In this paper, we also use Blue Waters traces. These traces can be acquired at https://bluewaters.ncsa.illinois.edu/data-sets ClusterWise Project ClusterWise is a Python project that contains all the code we wrote to generate and parse the traces. The project includes a README file with more information on how to use it. You can also check the `--help` option in the `clusterWise.py` file. The Data Tar File For the analyses conducted in the paper, we generated some intermediate files. For instance, the output of the FTIO tool when studying the periodicity of the I/O. These files can be found in the `data.tar.gz` file. Moreover, it also contains Jupyter notebooks and Python code used to analyze the data. FTIO Artifacts Reproducibility Several results presented in the paper were generated using the FTIO tool. To fully reproduce these artifacts, please follow the steps in the repository: FTIO Artifacts Repository.《深入剖析高性能计算（High Performance Computing, HPC）应用的时序I/O行为》[数据集] 本仓库包含发表于**IPDPS 2025**的论文《深入剖析高性能计算应用的时序I/O行为》所使用的代码与数据集。本仓库结构如下： 信号文件包含本研究使用的I/O追踪数据。 clusterwise-master目录包含用于生成与分析追踪数据的Python工具集。 data.tar归档文件包含从追踪数据中提取的部分数据，用于支撑论文中的各项研究；除数据外，还附带用于生成图表与开展分析的Python代码。 ## 信号归档文件 本研究的信号文件包含从两套高性能计算系统采集的I/O追踪数据： ### PlaFRIM PlaFRIM是法国国家信息与自动化研究所（Inria）波尔多中心的实验平台，配备192个计算节点与BeeGFS存储系统，该系统包含2个对象存储服务器（Object Storage Server, OSS），每台服务器搭载4个对象存储目标（Object Storage Target, OST），默认条带化数量为4，网络带宽为100 Gbps，峰值I/O性能约为12 GiB/s。本次数据采集周期为26个月（2022年5月至2024年7月），通过`beegfs-ctl`命令以每秒一次的频率采集按用户分组的带宽使用情况。 ### SDumont SDumont坐落于巴西国家科学计算实验室（LNCC），拥有分布于1134个节点的36472个计算核心，峰值性能达5.1 petaflops。其Lustre存储系统部署于10个对象存储服务器，每台服务器搭载1个对象存储目标，默认条带化数量为1，峰值I/O性能为30 GiB/s。本次数据采集使用Collectl工具，每15秒从每个计算节点采集一次数据，采集周期为12个月（2020年1月至2020年12月）。 ## 数据集格式 PlaFRIM与SDumont数据集文件均采用逗号分隔值（Comma-Separated Values, CSV）格式，包含三列：read（读操作）、write（写操作）与both（读写总和）。 read与write列代表执行的I/O操作，列内数值表示对应操作的带宽： - PlaFRIM：单位为MiB/s - SDumont：单位为KiB/s both列包含读与写操作的带宽总和。 文件内每一行代表一次时长为1秒的测量结果。数据集中的每个文件对应一个作业，并以唯一的文件ID作为标识。 ## 蓝水（Blue Waters）系统 本研究还使用了蓝水系统的I/O追踪数据，该数据集可从https://bluewaters.ncsa.illinois.edu/data-sets获取。 ## ClusterWise项目 ClusterWise是一个Python项目，包含我们编写的所有用于生成与解析追踪数据的代码。该项目附带README文件，详细说明了使用方法；你也可以查看`clusterWise.py`文件中的`--help`选项获取更多信息。 ## 数据归档文件 针对论文中的分析任务，我们生成了若干中间文件，例如用于研究I/O周期性的FTIO工具输出结果，这些文件均存放于`data.tar.gz`归档文件中。此外，该归档还包含用于数据分析的Jupyter笔记本与Python代码。 ## FTIO实验制品可复现性 论文中的多项结果均通过FTIO工具生成。如需完整复现这些实验制品，请按照仓库中《FTIO Artifacts Repository》的说明步骤操作。