Experimental Evaluation Methodology for The Era of No Steady Performance (Artifact)

Recent studies of virtual machine warm up have pointed out that even small deterministic microbenchmarks executed in tightly controlled circumstances often do not reach a steady state of peak performance. This impacts performance evaluation methodologies that focus on performance after warm up, because the lack of a steady state may violate common assumptions made when computing metrics such as the average performance or the confidence interval for that average. Our work examines the reported lack of steady state in the context of comparatively larger virtual machine workloads. We document and analyze similar lack of steady state and argue that it should be considered an inherent property of these workloads rather than a fault. We introduce an updated performance evaluation methodology for workloads whose execution exhibits segments of steady state performance separated by sudden performance changes. Using the Renaissance benchmark suite for the Java Virtual Machine, we show that the methodology can produce confidence intervals that miss the true performance over 20% less often than the existing methodologies. The main artifact archive long-runs-artifact.tar.gz 12GB, requires Podman or Docker SHA256 804787d19bb0577dbdcf3af64e2fb755fe3797a7a01f94a494585f55ddaeefee Optional data for anomaly investigation Complete archive data-four-hour-vm-log-external.tar.gz 70GB SHA256 247df8ec5727c5ec7b3453aaf0c5701dd06eacc3895987cc9d0b5bd082b825d6 Minimal subset data-four-hour-vm-log-minimal.tar.gz 992KB SHA256 eb46e57c6b33cca09309b28dcf06b485932674d8ce5a3d31d11827db8b7dc67b Optional data for code warm up profiling Complete archive data-warm-up-profile-external.tar.gz 85GB SHA256 f9fa7a79b9ecf44e30f530b614a0a8c0808a101e24618839fcdf9a6bcd2b8cd3 Minimal subset data-warm-up-profile-minimal.tar.gz 56MB SHA256 523fcd841a48d2945aa950bdf1ac6505ea5b2b438356b4e2082c95d732d05c23 The standalone computation library 9kB, requires R Library README longruns_README.md Library tarball longruns_0.0.0.tar.gz This version of the record contains only the minimal subset of the two optional data archives, the complete archives are available at https://doi.org/10.48700/datst.nx4g1-x9m42. Compared to the earlier versions, this version of the record contains updated package dependencies and updated replication instructions. No material changes were done to either the data or the computations.