Experimental Evaluation Methodology for The Era of No Steady Performance (Artifact)

Recent studies of virtual machine warm up have pointed out that even small deterministic microbenchmarks executed in tightly controlled circumstances often do not reach a steady state of peak performance. This impacts performance evaluation methodologies that focus on performance after warm up, because the lack of a steady state may violate common assumptions made when computing metrics such as the average performance or the confidence interval for that average. Our work examines the reported lack of steady state in the context of comparatively larger virtual machine workloads. We document and analyze similar lack of steady state and argue that it should be considered an inherent property of these workloads rather than a fault. We introduce an updated performance evaluation methodology for workloads whose execution exhibits segments of steady state performance separated by sudden performance changes. Using the Renaissance benchmark suite for the Java Virtual Machine, we show that the methodology can produce confidence intervals that miss the true performance over 20% less often than the existing methodologies. The main artifact archive long-runs-artifact.tar.gz 12GB, requires Podman or Docker SHA256 f6c23dde7661304d3bc0230f9a618629b66c89f29624c38796aa42b3fe217fcf Optional data for anomaly investigation Complete archive data-four-hour-vm-log-external.tar.gz 70GB SHA256 247df8ec5727c5ec7b3453aaf0c5701dd06eacc3895987cc9d0b5bd082b825d6 Minimal subset data-four-hour-vm-log-minimal.tar.gz 992KB SHA256 eb46e57c6b33cca09309b28dcf06b485932674d8ce5a3d31d11827db8b7dc67b Optional data for code warm up profiling Complete archive data-warm-up-profile-external.tar.gz 85GB SHA256 f9fa7a79b9ecf44e30f530b614a0a8c0808a101e24618839fcdf9a6bcd2b8cd3 Minimal subset data-warm-up-profile-minimal.tar.gz 56MB SHA256 523fcd841a48d2945aa950bdf1ac6505ea5b2b438356b4e2082c95d732d05c23 The standalone computation library 9kB, requires R Library README longruns_README.md Library tarball longruns_0.0.0.tar.gz This version of the record contains only the minimal subset of the two optional data archives.既往针对虚拟机预热（virtual machine warm up）的研究指出，即便在严格受控的实验环境下运行的小型确定性微基准测试（deterministic microbenchmarks），往往也无法达到峰值性能的稳态。这会对以预热后性能为评估目标的性能评估方法产生影响：由于缺乏稳态，计算平均性能或该平均值的置信区间（confidence interval）等指标时所依赖的通用假设将被打破。 本研究针对规模相对更大的虚拟机工作负载场景，对此前报道的稳态缺失现象展开分析。我们记录并分析了同类稳态缺失现象，并提出该现象应被视作此类工作负载的固有属性，而非程序故障。针对执行过程中存在多段稳态性能区间、且区间间存在性能突变的工作负载，我们提出了一套更新后的性能评估方法。通过采用面向Java虚拟机（Java Virtual Machine）的Renaissance基准测试套件（Renaissance benchmark suite）开展验证实验，我们证明该方法生成的置信区间，其真实性能漏判率较现有方法降低20%以上。 主工件存档文件 long-runs-artifact.tar.gz 大小：12GB，需依赖Podman或Docker SHA256：f6c23dde7661304d3bc0230f9a618629b66c89f29624c38796aa42b3fe217fcf 异常调查可选数据集 完整归档 data-four-hour-vm-log-external.tar.gz 大小：70GB SHA256：247df8ec5727c5ec7b3453aaf0c5701dd06eacc3895987cc9d0b5bd082b825d6 最小子集 data-four-hour-vm-log-minimal.tar.gz 大小：992KB SHA256：eb46e57c6b33cca09309b28dcf06b485932674d8ce5a3d31d11827db8b7dc67b 代码预热分析可选数据集 完整归档 data-warm-up-profile-external.tar.gz 大小：85GB SHA256：f9fa7a79b9ecf44e30f530b614a0a8c0808a101e24618839fcdf9a6bcd2b8cd3 最小子集 data-warm-up-profile-minimal.tar.gz 大小：56MB SHA256：523fcd841a48d2945aa950bdf1ac6505ea5b2b438356b4e2082c95d732d05c23 独立计算库 大小：9kB，需依赖R语言 库说明文档：longruns_README.md 库压缩包：longruns_0.0.0.tar.gz 本版本的记录仅包含两个可选数据归档文件的最小子集。