Data and analysis scripts for the OSDI 2018 paper "Taming Performance Variability"

This repository contains our raw datasets and notebooks for analyzing performance results of benchmarks executed on CloudLab machines. File organization: notebooks/ - Various Jupyter notebooks containing our analysis. Contains analysis on our main dataset as well as some analysis on one-off collections. bench-lib/common.py - Common python utilities that are shared by the notebooks. disk-process.ipynb - Main notebook for analysis of disk performance (both HDDs and SSDs). e-vs-cov.ipynb - Analysis of E vs CoV for selected configurations. env-test.ipynb - Version information for Python and installed packages. kernel-testing-Nd.ipynb - Analysis of data using kernel two-sample testing in one and multiple dimensions.. mem-process.ipynb - Main notebook for analysis of memory performance. network-process.ipynb - Main notebook for analysis of network performance. normality.ipynb - Notebook for the high-level analysis of normality (or lack of it) in the collected data. quantile-regression.ipynb - Notebook for Quantile Regression analysis for both disk and memory data. temporal-analysis.ipynb - Notebook to search for/analyze any temporal aspects to our dataset. testbed-coverage.ipynb - Analysis of the extent to which we were able to cover all of the test hardware. variability.ipynb - Notebook for the high-level analysis of variability. wisc-disktests.ipynb - ONE-OFF RUN: Analyzing variation in fio results on Wisconsin SSDs over repeated runs. wisc-memtests.ipynb - ONE-OFF RUN: Analysis of benchmark order for various Wisconsin hardware/memory configurations. data/. Contains a dump of the dataset (as of 04/04/2018). This dataset is generated by running the code contained in the https://gitlab.flux.utah.edu/emulab/cloudlab-orchestration repository. Some additional one-off collections are contained here as well. CoV-Summary/ - Data derived from our raw dataset for use by notebooks/e-vs-cov.ipynb. nodes/ - Simple list of all nodeids for each hardware type for use in notebooks/testbed-coverage.ipynb. raw-data/ - Raw .sql (and .csv files made from it) file containing a dump of our primary dataset through April 4th, 2018, used for the majority of the notebooks. Filtering is done in our notebooks to exclude data past April 1st, 2018 (as well as remove runs that appear to contain impossible execution conditions). wisc-disktests/ - ONE-OFF RUN: Results from successive runs of fio on Wisconsin SSDs. Used in notebooks/wisc-disktests.ipynb. wisc-memtests/ - ONE-OFF RUN: Results from memory benchmarks run with specific orderings on various Wisconsin machines. Used in notebooks/wisc-memtests.ipynb. wisc-pagemaps/ - ONE-OFF RUN: Virtual-to-Physical memory address mappings for tests run in wisc-memtests/. Used in notebooks/wisc-memtests.ipynb.本仓库包含用于分析在CloudLab机器上执行的基准测试性能结果的原始数据集与Jupyter笔记本（Jupyter Notebook）脚本。文件组织如下： notebooks/：存放各类Jupyter笔记本分析脚本，涵盖主数据集分析与单次临时采集数据的分析任务。 bench-lib/common.py：供所有Notebook共享使用的通用Python工具库。 disk-process.ipynb：针对磁盘性能（涵盖机械硬盘（Hard Disk Drive, HDD）与固态硬盘（Solid State Drive, SSD））开展分析的核心Notebook脚本。 e-vs-cov.ipynb：针对选定配置的E值与变异系数（Coefficient of Variation, CoV）开展分析的脚本。 env-test.ipynb：用于查看Python运行环境与已安装依赖包版本信息的脚本。 kernel-testing-Nd.ipynb：针对一维与多维核两样本检验数据开展分析的脚本。 mem-process.ipynb：针对内存性能开展分析的核心Notebook脚本。 network-process.ipynb：针对网络性能开展分析的核心Notebook脚本。 normality.ipynb：用于对采集数据的正态性（或非正态性）开展高层次分析的脚本。 quantile-regression.ipynb：针对磁盘与内存数据开展分位数回归分析的脚本。 temporal-analysis.ipynb：用于检索、分析数据集潜在时间特征的脚本。 testbed-coverage.ipynb：用于分析本次测试覆盖硬件范围的分析脚本。 variability.ipynb：用于对数据离散程度开展高层次分析的脚本。 wisc-disktests.ipynb：单次临时运行（ONE-OFF RUN）：分析威斯康星大学SSD上重复执行fio测试的结果差异。 wisc-memtests.ipynb：单次临时运行（ONE-OFF RUN）：针对各类威斯康星大学硬件/内存配置的基准测试执行顺序分析脚本。 data/：包含截至2018年4月4日的数据集快照。该数据集由https://gitlab.flux.utah.edu/emulab/cloudlab-orchestration仓库中的代码生成，同时还包含部分额外的单次临时采集数据集。 CoV-Summary/：从原始数据集导出的衍生数据，供notebooks/e-vs-cov.ipynb调用使用。 nodes/：按硬件类型分类的所有节点ID清单，供notebooks/testbed-coverage.ipynb调用使用。 raw-data/：包含截至2018年4月4日的主数据集结构化查询语言（Structured Query Language, SQL）文件（及由其导出的逗号分隔值（Comma-Separated Values, CSV）文件）快照，为绝大多数Notebook脚本所使用。所有Notebook会通过过滤操作排除2018年4月1日之后的测试数据，并移除存在异常执行条件的测试运行记录。 wisc-disktests/：单次临时运行（ONE-OFF RUN）：威斯康星大学SSD上重复执行fio测试的结果数据集，供notebooks/wisc-disktests.ipynb使用。 wisc-memtests/：单次临时运行（ONE-OFF RUN）：各类威斯康星大学主机上按特定顺序执行的内存基准测试结果数据集，供notebooks/wisc-memtests.ipynb使用。 wisc-pagemaps/：单次临时运行（ONE-OFF RUN）：wisc-memtests中测试任务的虚拟地址到物理地址映射数据数据集，供notebooks/wisc-memtests.ipynb使用。