Dataset (2025) for article "Resource Optimization with MPI Process Malleability for Dynamic Workloads in HPC Clusters"

This dataset was generated and used in the publication "Resource Optimization with MPI Process Malleability for Dynamic Workloads in HPC Clusters." The dataset is organized into three stages: raw data, preprocessed data, and processed data. Each workload execution includes log files, application code, executables, and launching scripts. Execution times are extracted from log files, specifically from "slurm-dmr_*.out" and "slurm-dmr_*.info", where "*" represents a number corresponding to a specific job execution. Dataset Structure: 1. raw_data Contains the output files from executing workloads on the MarenostrumV HPC cluster. This section is divided into three subsections, each corresponding to a different workload type: Static_Workload: Data for the static workload, which does not use malleability. Sync_Workload: Data for the synchronous dynamic workload, including results for both baseline and merge configurations (5 executions each). Async_Workload: Data for the asynchronous dynamic workload, including results for both baseline and merge configurations (5 executions each). 2. preprocessed_data This section contains the collected raw data in .pkl files, following the same structure as the raw_data folder. For each workload execution, four .pkl files are generated. The variable name can take values from [baseline, merge, static], while X represents a workload execution number: If X = J, the file contains a compilation of all workloads with the same configuration. If A appears before X, it refers to an asynchronous execution. The four types of .pkl files are: nameX_data.pkl: Contains application runtime data. A description is available in nameX_data_description.txt. nameX_data_resize.pkl: Contains application resize data. A description is available in nameX_data_resize_description.txt. nameAX_iter_data.pkl: Contains iteration time data for asynchronous (A) workloads. A description is available in nameAX_iter_data_description.txt. nameX_workload.pkl: Contains Slurm workload metrics. A description is available in nameX_workload_description.txt. 3. processed_data Includes the analyzed results from the preprocessed_data folder. This section contains .xlsx files and images used in the Experimental Setup section of the paper. The Excel files are categorized as follows: Exec_dataX.xlsx: Contains application execution results. Mall_dataX.xlsx: Contains resize time results for dynamic executions. 4. Codes This folder contains the scripts used to convert raw_data into preprocessed_data, along with a Jupyter Notebook used for data analysis and visualization. To understand or use these codes, please contact the dataset creators.本数据集生成并应用于发表论文《高性能计算集群动态工作负载下基于消息传递接口（MPI）进程可伸缩性的资源优化》（原标题：Resource Optimization with MPI Process Malleability for Dynamic Workloads in HPC Clusters）。数据集分为三个阶段：原始数据、预处理数据与处理后数据。 每份工作负载执行产物包含日志文件、应用代码、可执行文件与启动脚本。执行时长从日志文件中提取，具体来自`slurm-dmr_*.out`与`slurm-dmr_*.info`，其中`*`代表对应特定作业执行的编号。 数据集结构： 1. 原始数据（raw_data） 包含在MarenostrumV高性能计算（HPC）集群上运行工作负载所产生的输出文件。本模块分为三个子模块，分别对应不同的工作负载类型： - 静态工作负载（Static_Workload）：不使用可伸缩性的静态工作负载数据集。 - 同步动态工作负载（Sync_Workload）：同步动态工作负载数据集，包含基线配置与合并配置的运行结果（各5次执行）。 - 异步动态工作负载（Async_Workload）：异步动态工作负载数据集，包含基线配置与合并配置的运行结果（各5次执行）。 2. 预处理数据（preprocessed_data） 本模块包含以`.pkl`格式存储的整合原始数据，目录结构与raw_data文件夹保持一致。针对每一次工作负载执行，将生成4个`.pkl`文件。变量名可选值为`baseline`（基线）、`merge`（合并）与`static`（静态），`X`代表工作负载执行编号： - 若X=J，则该文件包含所有相同配置工作负载的整合数据。 - 若X前带有前缀`A`，则代表异步执行（A）的工作负载。 四类`.pkl`文件分别为： - `nameX_data.pkl`：包含应用运行时数据，相关说明详见`nameX_data_description.txt`。 - `nameX_data_resize.pkl`：包含应用伸缩调整数据，相关说明详见`nameX_data_resize_description.txt`。 - `nameAX_iter_data.pkl`：包含异步（A）工作负载的迭代时长数据，相关说明详见`nameAX_iter_data_description.txt`。 - `nameX_workload.pkl`：包含Slurm工作负载指标数据，相关说明详见`nameX_workload_description.txt`。 3. 处理后数据（processed_data） 包含从preprocessed_data文件夹中分析得到的结果，本模块包含论文实验部分所用的`.xlsx`文件与图像文件。Excel文件分类如下： - `Exec_dataX.xlsx`：包含应用执行结果数据。 - `Mall_dataX.xlsx`：包含动态执行的伸缩调整时长数据。 4. 代码（Codes） 本文件夹包含用于将raw_data转换为preprocessed_data的脚本，以及用于数据分析与可视化的Jupyter笔记本。如需理解或使用这些代码，请联系数据集创建者。