Data and analysis scripts for the OSDI 2018 paper "Taming Performance Variability"

This repository contains our raw datasets and notebooks for analyzing performance results of benchmarks executed on CloudLab machines. File organization: notebooks/ - Various Jupyter notebooks containing our analysis. Contains analysis on our main dataset as well as some analysis on one-off collections. bench-lib/common.py - Common python utilities that are shared by the notebooks. disk-process.ipynb - Main notebook for analysis of disk performance (both HDDs and SSDs). e-vs-cov.ipynb - Analysis of E vs CoV for selected configurations. env-test.ipynb - Version information for Python and installed packages. kernel-testing-Nd.ipynb - Analysis of data using kernel two-sample testing in one and multiple dimensions.. mem-process.ipynb - Main notebook for analysis of memory performance. network-process.ipynb - Main notebook for analysis of network performance. normality.ipynb - Notebook for the high-level analysis of normality (or lack of it) in the collected data. quantile-regression.ipynb - Notebook for Quantile Regression analysis for both disk and memory data. temporal-analysis.ipynb - Notebook to search for/analyze any temporal aspects to our dataset. testbed-coverage.ipynb - Analysis of the extent to which we were able to cover all of the test hardware. variability.ipynb - Notebook for the high-level analysis of variability. wisc-disktests.ipynb - ONE-OFF RUN: Analyzing variation in fio results on Wisconsin SSDs over repeated runs. wisc-memtests.ipynb - ONE-OFF RUN: Analysis of benchmark order for various Wisconsin hardware/memory configurations. data/. Contains a dump of the dataset (as of 04/04/2018). This dataset is generated by running the code contained in the https://gitlab.flux.utah.edu/emulab/cloudlab-orchestration repository. Some additional one-off collections are contained here as well. CoV-Summary/ - Data derived from our raw dataset for use by notebooks/e-vs-cov.ipynb. nodes/ - Simple list of all nodeids for each hardware type for use in notebooks/testbed-coverage.ipynb. raw-data/ - Raw .sql (and .csv files made from it) file containing a dump of our primary dataset through April 4th, 2018, used for the majority of the notebooks. Filtering is done in our notebooks to exclude data past April 1st, 2018 (as well as remove runs that appear to contain impossible execution conditions). wisc-disktests/ - ONE-OFF RUN: Results from successive runs of fio on Wisconsin SSDs. Used in notebooks/wisc-disktests.ipynb. wisc-memtests/ - ONE-OFF RUN: Results from memory benchmarks run with specific orderings on various Wisconsin machines. Used in notebooks/wisc-memtests.ipynb. wisc-pagemaps/ - ONE-OFF RUN: Virtual-to-Physical memory address mappings for tests run in wisc-memtests/. Used in notebooks/wisc-memtests.ipynb.