Replication Package for: Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud

Replication Package for: Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud

This is our replication package for our study on Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud.

All scalability experiments are performed with the scalability benchmarking framework Theodolite at Kiel University's Software Performance Engineering Lab (SPEL).

With this replication package, we provide:

Benchmark execution files in executions,

our benchmark (raw) results in results, and

analysis script for our results in analysis.

Repeating Benchmark Executions

All our Theodolite executions are tailored to either the SPEL cluster or the Google Cloud.

Kiel University's Software Performance Engineering Lab (SPEL)

The SPEL cluster consists of 5 nodes, named kube1-1 to kube1-5 and labeled with env=dev. To run them in your local cluster, make sure to provide the same infrastructure or rename node selectors in the execution files accordingly.

To install Theodolite, run:

helm install theodolite theodolite/theodolite --version 0.8.6 -f https://raw.githubusercontent.com/cau-se/theodolite/main/helm/preconfigs/extended-metrics.yaml -f se-cluster-dev.yaml

or for the vertical scalability experiment:

helm install theodolite theodolite/theodolite --version 0.8.6 -f https://raw.githubusercontent.com/cau-se/theodolite/main/helm/preconfigs/extended-metrics.yaml -f se-cluster-dev.yaml -f se-cluster-dev-vertical.yaml

See Theodolite's documentation for further usage instructions.

Google Cloud

In the public cloud baseline experiments, the cluster consists of 5 e2-standard-32 nodes.

To install Theodolite, run:

helm install theodolite theodolite/theodolite --version 0.8.6 -f https://raw.githubusercontent.com/cau-se/theodolite/main/helm/preconfigs/extended-metrics.yaml -f gcp-cluster-dev.yaml

For the experiments testing higher load intensities, the cluster consists of 4 e2-standard-16 nodes labeled with type=infra and 4 or 8 e2-standard-16 nodes with label type=sut. To install Theodolite in this cluster, run:

helm install theodolite theodolite/theodolite --version 0.8.6 -f https://raw.githubusercontent.com/cau-se/theodolite/main/helm/preconfigs/extended-metrics.yaml -f gcp-cluster-stress.yaml

In both cases, change the maximum load generated per load generator instance:

# Generate max. 100000 rec/sec per load generator instance
export MAX_RECORDS_PER_INSTANCE=100000
kubectl patch benchmarks uc1-beam-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]"
kubectl patch benchmarks uc1-beam-samza --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]"
kubectl patch benchmarks uc1-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]"
kubectl patch benchmarks uc1-hazelcastjet --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]"
kubectl patch benchmarks uc1-kstreams --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]"
kubectl patch benchmarks uc2-beam-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]"
kubectl patch benchmarks uc2-beam-samza --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]"
kubectl patch benchmarks uc2-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]"
kubectl patch benchmarks uc2-hazelcastjet --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]"
kubectl patch benchmarks uc2-kstreams --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]"
kubectl patch benchmarks uc3-beam-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]"
kubectl patch benchmarks uc3-beam-samza --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]"
kubectl patch benchmarks uc3-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]"
kubectl patch benchmarks uc3-hazelcastjet --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]"
kubectl patch benchmarks uc3-kstreams --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]"
kubectl patch benchmarks uc4-beam-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]"
kubectl patch benchmarks uc4-beam-samza --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]"
kubectl patch benchmarks uc4-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]"
kubectl patch benchmarks uc4-hazelcastjet --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]"
kubectl patch benchmarks uc4-kstreams --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]"

See Theodolite's documentation for further usage instructions.

Repeating Results Analysis

To inspect, repeat, or extend our results analysis, see results or run the corresponding notebooks in analysis.

For analyzing and visualizing benchmark results, either Docker or a Jupyter installation with Python 3.7 or 3.8 is required (e.g., in a virtual environment). Moreover, we require some Python libraries, which can be installed by:

pip install -r analysis/requirements.txt