Benchmarking the Overhead of Distributed Tracing Agents - Dataset

Contents This dataset contains the measurement results of our overhead measurement study "Benchmarking the Overhead of Distributed Tracing Agents" (https://doi.org/10.1145/3777884.3797004). By using this dataset, you can reproduce our measurements or try to repeat the measurements on different infrastructure (different CPU, OS, JVM version, …) in order to compare the results, or analyse our measurement results (mainly .csv files) in depth. It contains The state of the microbenchmark MooBench (commit e475465), the state with async profiler integrated (moobench-asprof.tar) and async profiler itself (async-profiler-4.1-linux-x64.tar.gz) The default parameter measurement results (default-results.tar.xz) The scalability experiments measurement results (exp-results-elasticapm-java.tar, exp-results-inspectIT-java.tar, exp-results-Kieker-java.tar, exp-results-OpenTelemetry-java.tar, exp-results-pinpoint-java.tar, exp-results-Scouter-java.tar, exp-results-Skywalking-java.tar) The flamegraph of the async-profiler analysis (flamegraphs.tar) Reproduction We executed three main experiments: Single user execution, scalability analysis, and time consumption analysis. The reproduction of these results is described in the following. Prerequisites The execution requires A Linux system, capable of running bash (our experiments were executed on Rocky Linux 9.6, other distributions are expected to work), including curl and awk A recent JDK installation (we used OpenJDK 21.0.8, OpenJDK 17 and above are expected to work) Docker 28.0.0 or newer for benchmarking Elastic APM, Skywalking, and Pinpoint (other frameworks can still be benchmarked without docker). Please make sure that the user that executes the experiments is able to execute the docker command (e.g., by adding the user to the docker group). A recent R installation (sudo dnf install epel-release; sudo dnf config-manager --set-enabled crb; sudo dnf install R) Before starting the experiments, do the following: Extract MooBench: tar -xvf moobench.tar.xz Build MooBench: cd moobench && ./setup.sh Set JAVA_JOME: export JAVA_HOME=.... Single User Execution To run the single user execution, execute the following steps (in the moobench-folder): cd frameworks export NUM_OF_LOOPS=30; ./benchmarkAll.sh (Number of loops can be set differently; our experiments were executed with 30 loops To obtain the table from the paper, move all result-* folders to a common folder, e.g., from the moobench home folder: mkdir single-user/; cp frameworks/*/results-* single-user/ Finally, execute cd ../analysis/; ./getStatistics.sh single-user/ The process is expected to take ~2 days on a recent machine. It can be sped up specifying a list of frameworks (e.g., ./benchmarkAll.sh inspectIT-java OpenTelemetry-java Kieker-java), or by reducing the number of loops. The file default-results.tar.xz contains our results of this experiment. Scalability Experiment To run the scalability benchmarks, execute the following steps (in the moobench-folder): cd frameworks export NUM_OF_LOOPS=30; ./runBenchmarkAll.sh ALL DEPTH (Number of loops can be adapted; instead of all, *one* framework name can be specified, e.g., inspectit-java) The process is expected to take multiple days per framework, so if it is executed for all 7 frameworks, the experiments need to be distributed. The files exp-results-*.tar.xz contain our results of this experiments. Time Consumption Analysis To execute the experiments, execute the following steps: Extract the adapted moobench version: tar -xvf moobench-asprof.tar && cd moobench && ./setup.sh Extract async profiler (we used version 4.1; it is expected that this also works with other versions, but for archival purposes, we uploaded version 4.1 here): tar -xvf async-profiler-4.1-linux-x64.tar.gz and export the path: export ASYNC_PROFILER_HOME=$(pwd)/async-profiler-4.1-linux-x64 Run the benchmark:cd moobench/framework && ./runAllWithAsyncProfiler.sh In every framework folder, individual results can be plotted, e.g., for Kieker using $ASYNC_PROFILER_HOME/bin/jfrconv flamegraph_1_10_5.collapsed flamegraph.html To get the mapping to time consumption sources, execute inside the moobench-folder: cd tools/compile-results/build/distributions/ && tar -xvf compile-results.tar && cd compile-results/lib; java -cp "$(printf '%s:' *.jar)" moobench.tools.flamegraphs.ClassifyNodes $DATAFOLDER/flamegraph_X.collapsed where $DATAFOLDER/flamegraph_X.collapsed points to the file you want to analyze; please note that performance measurements will differ depending on the speed of your hard disk etc., so while the general tendendency is expected the same, you will most likely see gradually different values. The file flamegraphs.tar.xz contain our results.