[Opendc-Sc18-Dataset] A Reference Architecture For Datacenter Scheduling: Data Artifacts

This release contains the data artifacts of the paper A Reference Architecture for Datacenter Scheduling presented at Supercomputing 2018 For the paper, experiments have been run on the following traces: <strong>Askalon (W-Eng)</strong> - <code>askalon_workload_ee</code> <strong>Chronos (W-Ind)</strong> - <code>chronos_exp_noscaler_ca</code> Each of the directories for the traces have the following structure: <strong>/setup.txt</strong><br> This text file describes the trace used for the experiment in addition to the amount of times the experiment was repeated and the amount of warm-up experiments. <strong>/setup.json</strong><br> This JSON file describes the topology of the datacenter used in the experiments. Each item represents the identifiers of the resource (here, CPU type) to use in the machine. The available CPU types are (1) Intel i7 (4 cores, 4100 MHz) and (2) Intel i5 (2 cores, 3500 MHz). <strong>/trace</strong><br> This directory contains the trace used in the simulation. The trace is stored in the Grid Workload Format. See the Grid Workload Archive for more information. <strong>/data/experiments.csv</strong><br> A CSV file containing information of all simulations that have been run on the OpenDC platform for this experiment. <strong>/data/job_metrics.csv</strong><br> A CSV file containing metrics (NSL, JMS, etc.) for each job that ran during the simulations. <strong>/data/stage_measurements.csv</strong><br> A CSV file containing timing measurements for the scheduling stages that ran during the simulations. <strong>/data/task_metrics.csv</strong><br> A CSV file containing metrics for each task that ran during the simulations. <strong>/data/tasks.csv</strong><br> A CSV file containing information about the tasks (submit time, runtime, etc.) that ran during the simulations as extracted from the traces. Additionally, we describe the format of each data file in the associated metadata file. <strong>Hardware</strong> The hardware used for running the experiments is a MacBook Pro with a 2,9 GHz Intel Core i7 processor and 16 GB 2133 MHz LPDDR3 internal memory. <strong>Reproduction</strong> This section describes the instructions for reproducing the paper results using a provided Docker image. Please make sure you have Docker installed and running. For reproduction, you will run the following experiments: <code>askalon_workload_ee</code><br> This is the large experiment of the paper and will take approximately 4 hours to complete similar hardware. <code>chronos_exp_noscaler_ca</code><br> This is the smaller experiment of the paper and will take approximately 5 minutes to complete on similar hardware. The Docker image <code>atlargeresearch/sc18-experiment-runner</code> can be used for running the experiments. A volume can be attached to the directory <code>/home/gradle/simulator/data</code> to capture the results of the experiments. Make sure you have, in your current working directory, the following files: <strong>/setup.json</strong><br> This JSON file describes the topology of the datacenter and can be found in this archive at <code>askalon_workload_ee/setup.json</code>. <strong>/askalon_workload_ee.gwf</strong><br> This file contains the trace for the Askalon workload. This file can be found in the archive at <code>askalon_workload_ee/trace/askalon_workload_ee.gwf</code>. <strong>/chronos_exp_noscaler_ca.gwf</strong><br> This file contains the trace for the Chronos workload. This file can be found in the archive at <code>chronos_exp_noscaler_ca/trace/chronos_exp_noscaler_ca.gwf</code>. Then, you can start the Askalon experiments as follows: <pre><code>$ docker run -it --rm -v $(pwd):/home/gradle/simulator/data atlargeresearch/sc18-experiment-runner -r 32 -w 4 -s data/setup.json data/askalon_workload_ee.gwf </code></pre> The experiment runner can be configured with the following options <strong>-r</strong>, <strong>--repeat</strong><br> The amount of times to repeat an experiment for each scheduler. <strong>-w</strong>, <strong>--warm-up</strong><br> The amount of times to warm-up the simulator for each scheduler. <strong>-p</strong>, <strong>--parallelism</strong><br> The number of experiments to run in parallel. <strong>--schedulers</strong><br> The list of schedulers to test, separated by spaces. The following schedulers are available: <code>SRTF-BESTFIT</code>, <code>SRTF-FIRSTFIT</code>, <code>SRTF-WORSTFIT</code>, <code>FIFO-BESTFIT</code>, <code>FIFO-FIRSTFIT</code>, <code>FIFO-WORSTFIT</code>, <code>RANDOM-BESTFIT</code>, <code>RANDOM-FIRSTFIT</code>, <code>RANDOM-WORSTFIT</code>. After the Askalon experiments have been finished, you can start the Chronos experiments. <strong>Make sure</strong> you have a copy of the result files in your directory as the result files will be overwritten. <pre><code>$ docker run -it --rm -v $(pwd):/home/gradle/simulator/data atlargeresearch/sc18-experiment-runner -r 32 -w 4 -s data/setup.json data/chronos_exp_noscaler_ca.gwf </code></pre>