Replication Package for: Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud

<strong>Replication Package for: Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud</strong> This is our replication package for our study on <em>Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud</em>. All scalability experiments are performed with the scalability benchmarking framework Theodolite at Kiel University's Software Performance Engineering Lab (SPEL). With this replication package, we provide: Benchmark execution files in <em>executions</em>, our benchmark (raw) results in <em>results</em>, and analysis script for our results in <em>analysis</em>. <strong>Repeating Benchmark Executions</strong> All our Theodolite executions are tailored to either the SPEL cluster or the Google Cloud. <em>Kiel University's Software Performance Engineering Lab (SPEL)</em> The SPEL cluster consists of 5 nodes, named <em>kube1-1</em> to <em>kube1-5</em> and labeled with <em>env=dev</em>. To run them in your local cluster, make sure to provide the same infrastructure or rename node selectors in the execution files accordingly. To install Theodolite, run: <pre><code class="language-bash">helm install theodolite theodolite/theodolite --version 0.8.6 -f https://raw.githubusercontent.com/cau-se/theodolite/main/helm/preconfigs/extended-metrics.yaml -f se-cluster-dev.yaml</code></pre> or for the vertical scalability experiment: <pre><code class="language-bash">helm install theodolite theodolite/theodolite --version 0.8.6 -f https://raw.githubusercontent.com/cau-se/theodolite/main/helm/preconfigs/extended-metrics.yaml -f se-cluster-dev.yaml -f se-cluster-dev-vertical.yaml</code></pre> See Theodolite's documentation for further usage instructions. <em>Google Cloud</em> In the public cloud baseline experiments, the cluster consists of 5 e2-standard-32 nodes. To install Theodolite, run: <pre><code class="language-bash">helm install theodolite theodolite/theodolite --version 0.8.6 -f https://raw.githubusercontent.com/cau-se/theodolite/main/helm/preconfigs/extended-metrics.yaml -f gcp-cluster-dev.yaml</code></pre> For the experiments testing higher load intensities, the cluster consists of 4 e2-standard-16 nodes labeled with <em>type=infra</em> and 4 or 8 e2-standard-16 nodes with label <em>type=sut</em>. To install Theodolite in this cluster, run: <pre><code class="language-bash">helm install theodolite theodolite/theodolite --version 0.8.6 -f https://raw.githubusercontent.com/cau-se/theodolite/main/helm/preconfigs/extended-metrics.yaml -f gcp-cluster-stress.yaml</code></pre> In both cases, change the maximum load generated per load generator instance: <pre><code class="language-bash"># Generate max. 100000 rec/sec per load generator instance export MAX_RECORDS_PER_INSTANCE=100000 kubectl patch benchmarks uc1-beam-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc1-beam-samza --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc1-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc1-hazelcastjet --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc1-kstreams --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc2-beam-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc2-beam-samza --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc2-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc2-hazelcastjet --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc2-kstreams --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc3-beam-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc3-beam-samza --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc3-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc3-hazelcastjet --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc3-kstreams --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc4-beam-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc4-beam-samza --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc4-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc4-hazelcastjet --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc4-kstreams --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]"</code></pre> See Theodolite's documentation for further usage instructions. <strong>Repeating Results Analysis</strong> To inspect, repeat, or extend our results analysis, see <em>results</em> or run the corresponding notebooks in <em>analysis</em>. For analyzing and visualizing benchmark results, either Docker or a Jupyter installation with Python 3.7 or 3.8 is required (e.g., in a virtual environment). Moreover, we require some Python libraries, which can be installed by: <pre><code class="language-bash">pip install -r analysis/requirements.txt</code></pre>