Understanding the Energy Consumption of Cloud-native Software Systems

These artifacts contain the dataset generated in the paper "Understanding the Energy Consumption of Cloud-native Software Systems". The dataset contains resource utilisation, power consumption, estimated power consumption and load metrics for a cloud-native software system. The test setup is consists of 6 machines running an OpenStack cluster. This OpenStack cluster is running 12 virtual machines that are in turn hosting a Kubernetes cluster. Performance and (estimated) power consumption are measured on all levels of the system.

The Bare-Metal (BM) and Virtual Machines (VM) are identified by their IP addresses. For BM the IP mapping is as follows:

192.168.1.109 - i3-04

192.168.1.110 - i3-02

192.168.1.111 - i5-02

192.168.1.112 - i3-01

192.168.1.113 - i5-01

192.168.1.114 - i5-04

For VM the IP mapping is:

192.168.1.190 - kubernetes-agent-7

192.168.1.191 - kubernetes-agent-6

192.168.1.192 - kubernetes-agent-10

192.168.1.194 - kubernetes-master

192.168.1.196 - kubernetes-agent-3

192.168.1.197 - kubernetes-agent-1

192.168.1.198 - kubernetes-agent-2

192.168.1.201 - kubernetes-agent-4

192.168.1.203 - kubernetes-agent-9

192.168.1.204 - kubernetes-agent-5

192.168.1.207 - kubernetes-agent-8

192.168.1.209 - kubernetes-agent-11

The VMs are deployed on the BMs as follows:

i3-01

kubernetes-agent-2

kubernetes-agent-8

i3-02

kubernetes-agent-1

kubernetes-agent-6

i3-04

kubernetes-agent-10

kubernetes-agent-11

i5-01

kubernetes-agent-3

kubernetes-agent-5

i5-02

kubernetes-master

kubernetes-agent-7

i5-04

kubernetes-agent-4

kubernetes-agent-9

Note that the following BMs are excluded from the experiments as they have other roles in the cluster and do not run workloads:

i3-03 (Juju - deploying OpenStack on BM nodes)

i3-05 (ProxMox - external observability tools that do not run in the cluster)

i5-03 (MAAS - provisioning of BM nodes)

Data Sets

The artifacts consists of 3 separate data sets: constant, direct and linear, corresponding to the respective load profile applied to the SUT as discussed in the paper. Each data set contains the same metrics, but the system is put under a different load.

Every experiment is repeated 3 times. All 3 repetitions are included in the data set. The data is collected using Prometheus and stored in JSON files.

For the constant and linear data sets, load is applied by deploying the "OpenTelemetry demo application", and sending automated user requests to the application. For the direct dataset, this application is not used and instead Kubernetes pods are created that apply a constant load to the cluster.

The timestamps of the experiments are as follows:

Run
Iteration
Start
End

Constant - 0 users
1
02-05-2024 10:08
02-05-2024 10:52

2
06-05-2024 13:20
06-05-2024 13:58

3
07-05-2024 09:20
07-05-2024 09:54

Constant - 50 users
1
02-05-2024 13:26
02-05-2024 13:57

2
15-05-2024 14:06
15-05-2024 14:38

3
07-05-2024 09:56
07-05-2024 10:39

Constant - 100 users
1
02-05-2024 14:00
02-05-2024 14:40

2
06-05-2024 14:40
06-05-2024 15:30

3
07-05-2024 10:42
07-05-2024 11:33

Constant - 200 users
1
16-05-2024 09:37
16-05-2024 10:09

2
06-05-2024 15:32
06-05-2024 16:13

3
07-05-2024 11:34
07-05-2024 12:11

Constant - 400 users
1
02-05-2024 15:24
02-05-2024 15:55

2
06-05-2024 16:14
06-05-2024 16:49

3
07-05-2024 12:13
07-05-2024 13:08

Linear
1
22-05-2024 15:13
22-05-2024 15:56

2
22-05-2024 16:05
22-05-2024 16:48

3
23-05-2024 09:58
23-05-2024 10:41

Direct
1
2024-05-23 13:01:28
2024-05-23 14:00:05

2
2024-05-23 14:33:23
2024-05-23 15:32:01

3
2024-05-23 15:39:36
2024-05-23 16:38:12

A complete log of the experiments can be found in experiments_log.txt.

Constant

The constant data set contains the metrics for the system under a constant load. The load is generated by a Locust script that sends requests to the system. Data collection starts 1 minute after the desired number of concurrent users is reached and requests have stabilised. This experiment is performed for the following constant number of concurrent users:

0 users

50 users

100 users

200 users

400 users

Note that the 0 users data does not contain the report_*.html and request_*.csv files, as these are generated by Locust and Locust is not run for the 0 users scenario.

Furthermore, note that Horizontal Pod Autoscaling is not enabled for this data set.

Linear

The linear dataset contains the metrics for the system under a linearly scaling load. The load is generated through Locust. It starts at 0 users, and scales up to 100 users at a rate of 1 user per 20 seconds. After the load reaches 100 users, another 10 minutes of data is recorded. Every dataset is 45 minutes long, with 1.7 minutes of no load, 33.3 minutes of scaling up, and 10 minutes of max load.

Note that Horizontal Pod Autoscaling is enabled for this data set.

Direct

The direct data set compliments the linear dataset. While the linear dataset provides a realistic load with things like networking factors being taken into account, the linear dataset can bottleneck on things like networking and the request client, so CPU usage is not maxed out. The direct dataset scales up linearly by applying direct CPU load to the Kubernetes pods without any application simulating a real usecase. The dataset works by deploying Kubernetes pods that max out immediately on exactly 200mCPU of load. The experiment starts with 1 pod, and 2 pods are added every 90 seconds up till 77 pods (the maximum number of pods the cluster allows to be scheduled). The first 1.5 minutes is no load, then 57 minutes to scale up, and then another 1.5 minutes at max load.

Note that the direct data set does not contain the app_*.json files, as this data set does not deploy an application and therefore no application specific metrics are collected. Instead, a script_log.txt is provided that explains when and how the direct load was scaled up.

Files

A table for each file in the data set can be found below, including the unit of the metric and a description of what data is collected in that file. The format of all JSON files is the API response format used by Prometheus. More information on this topic can be found here: https://prometheus.io/docs/prometheus/latest/querying/api/. All timestamps are in the CEST timezone.

Filename
Unit
Description

app_ads_ad_requests_total.json
Total Count
Total requests received by the ad microservice

app_currency_counter_total.json
Total Count
Total currency that circulated through the system

app_frontend_requests_total.json
Total Count
Total requests received by the frontend service

app_payment_transactions_total.json
Total Count
Total transactions made to the transaction microservice

app_recommendations_counter_total.json
Total Count
Total recommendations made by the recommendation microservice

container_blkio_device_usage_total.json
Total Bytes
Total bytes used by blkio devices for pods

container_cpu_usage_seconds_total.json
Total Seconds
Cumulative cpu time consumed by the pod

container_cpu_user_seconds_total.json
Total Seconds
Cumulative user cpu time consumed by the pod

container_fs_reads_bytes_total.json
Total Bytes
Cumulative count of bytes read by the pod

container_fs_writes_bytes_total.json
Total Bytes
Cumulative count of bytes written by the pod

container_memory_rss.json
Bytes
Resident Set Size of the pod

kepler_container_bpf_cpu_time_ms_total.json
Milliseconds
CPU time for the pod as measured through a Kepler BPF program

kepler_container_core_joules_total.json
Total Joules
Total energy consumption of CPU cores used by a pod

kepler_container_dram_joules_total.json
Total Joules
Total energy consumption of DRAM used by a pod

kepler_container_joules_total.json
Total Joules
Aggregated total energy consumption of a pod

kepler_container_package_joules_total.json
Total Joules
Cumulative energy consumed by all cores and uncore components of a pod

kepler_node_core_joules_total.json
Total Joules
Aggregation of core_joules of all pods running on a Kubernetes node

kepler_node_dram_joules_total.json
Total Joules
Aggregation of dram_joules of all pods running on a Kubernetes node

kepler_node_package_joules_total.json
Total Joules
Aggregation of package_joules of all pods running on a Kubernetes node

node_cpu_scaling_frequency_hertz.json
Hertz
Current scaled cpu thread frequency of a machine (BM or VM)

node_cpu_seconds_total.json
Total Seconds
Total number of seconds the CPU worked on a machine (BM or VM)

node_disk_read_time_seconds_total.json
Total Seconds
Total number of seconds spent reading disk on a machine (BM or VM)

node_disk_write_time_seconds_total.json
Total Seconds
Total number of seconds spent writing disk on a machine (BM or VM)

node_hwmon_temp_celsius.json
Celsius
Temperature of the machine (BM) as reported by its monitoring hardware

node_load1.json
Load Average
Load on the machine (BM or VM) averaged over 1 minute

node_load5.json
Load Average
Load on the machine (BM or VM) averaged over 5 minutes

node_load15.json
Load Average
Load on the machine (BM or VM) averaged over 15 minutes

node_memory_Active_bytes.json
Bytes
Active number of bytes in memory on the machine (BM or VM)

node_memory_Committed_AS_bytes.json
Bytes
Committed number of bytes in memory on the machine (BM or VM)

node_rapl_core_joules_total.json
Total Joules
Total energy consumption of CPU cores by a machine, estimated by RAPL

node_rapl_dram_joules_total.json
Total Joules
Total energy consumption of DRAM by a machine, estimated by RAPL

node_rapl_package_joules_total.json
Total Joules
Total energy consumption of the machine package, estimated by RAPL

node_rapl_psys_joules_total.json
Total Joules
Total energy consumption of the machine psys, estimated by RAPL

power_consumption.json
Watt
Energy consumption as measured by the physical power plugs

report_*.html
-
HTML report describing details of locust actions during the experiment

requests_*.csv
-
Request summary per endpoint generated by locust

scaph_host_power_microwatts.json
Microwatt
Power consumption of the whole machine as estimated by Scaphandre

scaph_process_cpu_usage_percentage.json
Percentage
Per-process CPU usage as a percentage of total machine CPU

scaph_process_memory_bytes.json
Bytes
Per-process memory usage

scaph_process_power_consumption_microwatts.json
Microwatt
Per-process energy consumption as estimated by Scaphandre

script.log
-
Log for the direct experiments for scaling up the pods