TOTO performance evaluation on Irene (Lustre) with IOR

This dataset contains the measurements used to evaluate TOTO, a transparent I/O tuning tool that dynamically selects the Lustre stripe count of files at runtime, on the Joliot Curie (Irene) supercomputer, using the IOR benchmarking tool to emulate a range of HPC access patterns. TOTO is open-source and available at: https://gitlab.inria.fr/hpc_io/toto For each access pattern, the dataset contains runs in three regimes: TOTO (dynamic): TOTO is enabled and selects the stripe count at runtime. Static, stripe count = 1: IOR runs without TOTO, with the Lustre default stripe count on Irene (1). Static, stripe count = nperf: IOR runs without TOTO, with the stripe count that we found to maximize performance for that pattern (nperf). Comparing these three regimes on the same patterns is what allows quantifying the benefit (or overhead) of TOTO with respect to both the system default and the best static configuration. The dataset was used in the experiments reported in [1]. All experiments were carried out on the Joliot Curie (Irene) supercomputer at the Très Grand Centre de Calcul (TGCC), operated by CEA, between 2024 and 2025, on the AMD-Rome partition. The Lustre file system on Irene is deployed with 1 MDS+MDT and 40 OSSs, with one OST per OSS, and a default stripe size of 1 MiB. Experiments scaled up to 8192 MPI processes on 64 nodes. Experiments were generated and orchestrated using the IOPS framework, an open-source tool that automates the description, execution, and post-processing of HPC I/O benchmarking campaigns: https://iops.gitlabpages.inria.fr/ IOR version 4.1.0+dev was used, with the POSIX API (-a POSIX). File format The dataset is provided as a single CSV file (results_zenodo.csv) in text format. Each row corresponds to one repetition of one experiment configuration. Multiple repetitions of each configuration were executed (at least 3 per configuration). Repetitions were submitted in random order, so the value of the execution.repetition column does not carry any ordering information beyond differentiating between repetitions of the same configuration. The dataset is organized around two regimes: TOTO runs: rows with vars.with_toto = True. TOTO is enabled and selects the Lustre stripe count of files at runtime. Baseline runs: rows with vars.with_toto = False. IOR runs without TOTO, with a fixed stripe count. The vars.baseline_stripe column indicates which baseline is being measured: one (stripe count = 1, the Lustre default on Irene) or nperf (stripe count = nperf, the best-performing static value we identified for the pattern). Columns Configuration variables (vars.*): vars.with_toto: True for TOTO runs, False for baseline runs. vars.baseline_stripe: for baseline runs, identifies which static baseline is being measured (one or nperf). Not meaningful for TOTO runs. vars.nodes: number of compute nodes used. vars.total_procs: total number of MPI processes used (across all nodes). vars.procs_per_node: number of processes per node. vars.cores_per_tasks: number of cores per MPI task. vars.filestrategy: either shared-file (a single file accessed by all processes) or file-per-proc (each process has its own file, IOR option -F). vars.spatiality: contig (each process accesses a contiguous portion of the file), strided (1D-strided access pattern, only with shared-file), or random (IOR option -z). vars.request_size_kb: IOR request size in KiB (IOR option -t). vars.block_size_kb: IOR block size per process in KiB (IOR option -b). vars.segments: IOR number of segments (IOR option -s). vars.volume_size_gb: total amount of data accessed in the experiment, in GiB. The amount accessed per process (for contig patterns) is therefore vars.volume_size_gb / vars.total_procs. vars.stripe_count: the Lustre stripe count actually used for the experiment (the number of OSTs across which files are striped). For baseline runs this is the configured value; for TOTO runs this is the initial value used by TOTO at file creation (TOTO may then place data on files with different stripe counts during execution). vars.nperf: the stripe count that maximizes performance for the access pattern (used as the nperf static baseline). vars.analysis_period: the analysis period configured for TOTO, in milliseconds. Only meaningful for vars.with_toto = True; equals 0 for baseline runs. Execution metadata: benchmark.name: name of the IOPS benchmark definition (toto-irene). execution.execution_id: identifier of an experiment configuration. All repetitions of the same configuration share the same execution_id. execution.repetition: repetition index within an execution_id. metadata.jobid: SLURM job id. metadata.executor_status, metadata.slurm_state, metadata.slurm_exitcode: execution status as reported by IOPS and SLURM. All published rows are SUCCEEDED / COMPLETED. metadata.submission_time, metadata.end: SLURM submission and end timestamps for the job. metadata.sysinfo.hostname, metadata.sysinfo.cpu_model, metadata.sysinfo.cpu_cores, metadata.sysinfo.memory_kb, metadata.sysinfo.kernel, metadata.sysinfo.os, metadata.sysinfo.ib_devices, metadata.sysinfo.filesystems: information about the compute node on which the run was scheduled. metadata.sysinfo.duration_seconds: wall-clock duration reported by the IOPS executor, in seconds. Performance metrics: metrics.write_bwMiB: aggregated write bandwidth reported by IOR, in MiB/s. metrics.write_totalTime: total write time reported by IOR (including open and close), in seconds. metrics.read_bwMiB: aggregated read bandwidth reported by IOR, in MiB/s. metrics.read_totalTime: total read time reported by IOR (including open and close), in seconds. How to use Each access pattern is described by the tuple of configuration variables that defines an IOR workload, namely: (vars.nodes, vars.total_procs, vars.procs_per_node, vars.filestrategy, vars.spatiality, vars.request_size_kb, vars.block_size_kb, vars.segments, vars.volume_size_gb, vars.nperf) For a given pattern, the dataset typically contains three groups of rows: vars.with_toto = True: TOTO selects the stripe count dynamically. vars.with_toto = False and vars.baseline_stripe = one: static stripe count 1 (the Lustre default on Irene). vars.with_toto = False and vars.baseline_stripe = nperf: static stripe count equal to the best-performing value for the pattern. A typical analysis aggregates each of these groups across repetitions (execution.repetition) using metrics.write_bwMiB and metrics.read_bwMiB, and then compares the three regimes. Median across repetitions is recommended given the variability observed on the production system. References [1] Francieli Boito, Luan Teylo, Mihail Popov, Laora Aimi, Alexis Bandet, Laércio Lima Pilla, Guillaume Pallez. TOTO: Transparent I/O Tuning for HPC Applications. ACM International Conference on Supercomputing (ICS), July 2026, Belfast, Northern Ireland, United Kingdom. [2] IOPS, an open-source framework for describing, running, and post-processing HPC I/O benchmarking campaigns. https://iops.gitlabpages.inria.fr/ [3] TOTO source code repository: https://gitlab.inria.fr/hpc_io/toto