Replication Package for "TriggerBench: A Performance Benchmark for Serverless Function Triggers"

This replication package contains the code (`aws-triggers` and `azure-trigger`), data analysis scripts (`data-analysis`), and dataset (`data`) of the TriggerBench cross-provider serverless benchmark. It also bundles a customized extension of the `serverless-benchmarker` tool to automate and analyze serverless performance experiments. <strong>TriggerBench</strong> The Github repository joe4dev/trigger-bench contains the last version of TriggerBench. This replication package describes the version for the paper "TriggerBench: A Performance Benchmark for Serverless Function Triggers". TriggerBench currently supports three triggers on AWS and eight triggers on Microsoft Azure. <strong>Dataset</strong> The `data/aws` and `data/azure` directories contain data from benchmark executions from April 2022. Each execution is a separate directory with a timestamp in the format `yyyy-mm-dd-HH-MM-SS` (e.g., `2022-04-15_21-58-52`) and contains the following files: `k6_metrics.csv`: Load generator HTTP client logs in CSV format (see [K6 docs](https://k6.io/docs/results-visualization/csv/)) `sb_config.yml`: serverless benchmarker execution configuration including experiment label. `trigger.csv`: analyzer output CSV per trace. `root_trace_id`: The trace id created by k6 and adopted by the invoker function `child_trace_id`: The trace id newly created by the receiver function if trace propagation is not supported (this is the case for most asynchronous triggers) `t1`-`t4`: Timestamps following the trace model (see paper) `t5`-`t9`: Additional timestamps for measuring timestamping overhead `coldstart_f1=True|False`: coldstart status for invoker (f1) and receiver (f2) functions `trace_ids.txt`: text file with each pair of `root_trace_id` and `child_trace_id` on a new line. `traces.json`: raw trace JSON representation as retrieved from the provider tracing service. For AWS, see [X-Ray segment docs](https://docs.aws.amazon.com/xray/latest/devguide/xray-api-segmentdocuments.html). For Azure, see [Application Insights telemetry data model](https://docs.microsoft.com/en-us/azure/azure-monitor/app/data-model). `workload_options.json`: [K6 load scenario](https://k6.io/docs/using-k6/scenarios/) configuration. <strong>Replicate Data Analysis</strong> <strong>Installation</strong> 1. Install [Python](https://www.python.org/downloads/) 3.10+ 2. Install Python dependencies `pip install -r requirements.txt` <strong>Create Plots</strong> 1. Run `python plots.py` generates the plots and the statistical summaries presented in the paper. By default, the plots will be saved into a `plots` sub-directory.<br> An alternative output directory can be configured through the environment variable `PLOTS_PATH`. &gt; Hint: For interactive development, we recommend the VSCode [Python extension](https://marketplace.visualstudio.com/items?itemName=ms-python.python) in [interactive mode](https://youtu.be/lwN4-W1WR84?t=107). <strong>Replicate Cloud Experiments</strong> The following experiment plan automates benchmarking experiments with different types workloads (constant and bursty). This generates a new dataset in the same format as described above. Set up a load generator as vantage point following the description in [LOADGENERATOR](./serverless-benchmarker/docs/LOADGENERATOR.md). Choose the `PROVIDER` (aws or azure) in the [constant.py](./experiment-plans/constant.py) experiment plan Run the [constant.py](./experiment-plans/constant.py) experiment plan Open tmux Activate virtualenv `source sb-env/bin/activate` Run `./constant.py 2&gt;&amp;1 | tee -a constant.log`本复现包包含了TriggerBench跨厂商无服务器基准测试的代码（`aws-triggers`与`azure-trigger`）、数据分析脚本（`data-analysis`）以及数据集（`data`）。同时还集成了`serverless-benchmarker`（无服务器基准测试工具）的定制化扩展版本，用于自动化开展无服务器性能实验并进行分析。 **TriggerBench** 官方GitHub仓库joe4dev/trigger-bench存储了TriggerBench的最新版本。本复现包对应论文《TriggerBench：无服务器函数触发器性能基准测试》所使用的版本。目前TriggerBench支持AWS平台的3种触发器，以及微软Azure平台的8种触发器。 **Dataset** `data/aws`与`data/azure`目录存储了2022年4月的基准测试执行数据。每一次执行对应一个以`yyyy-mm-dd-HH-MM-SS`格式命名的时间戳子目录（例如`2022-04-15_21-58-52`），其中包含以下文件： - `k6_metrics.csv`：以CSV格式存储的负载生成器HTTP客户端日志（详见[K6官方文档](https://k6.io/docs/results-visualization/csv/)） - `sb_config.yml`：无服务器基准测试工具执行配置文件，包含实验标签 - `trigger.csv`：针对每条追踪记录的分析器输出CSV文件 - `root_trace_id`：由k6生成并被调用函数（f1）采用的追踪ID - `child_trace_id`：若不支持追踪传播（多数异步触发器均存在此情况），则由接收函数（f2）新生成的追踪ID - `t1`-`t4`：遵循论文中追踪模型的时间戳 - `t5`-`t9`：用于测量时间戳开销的额外时间戳 - `coldstart_f1=True|False`：调用函数（f1）与接收函数（f2）的冷启动状态 - `trace_ids.txt`：文本文件，每行存储一组`root_trace_id`与`child_trace_id` - `traces.json`：从云厂商追踪服务获取的原始追踪JSON数据。AWS平台相关内容详见[AWS X-Ray分段文档](https://docs.aws.amazon.com/xray/latest/devguide/xray-api-segmentdocuments.html)；Azure平台相关内容详见[Azure应用程序见解遥测数据模型](https://docs.microsoft.com/en-us/azure/azure-monitor/app/data-model) - `workload_options.json`：[K6负载场景](https://k6.io/docs/using-k6/scenarios/)配置文件 **复现数据分析流程** **安装步骤** 1. 安装[Python](https://www.python.org/downloads/) 3.10及以上版本 2. 安装Python依赖包：执行`pip install -r requirements.txt` **生成图表** 1. 运行`python plots.py`即可生成论文中展示的图表与统计摘要。默认情况下，图表将保存至`plots`子目录。也可通过环境变量`PLOTS_PATH`配置自定义输出目录。 > 提示：若进行交互式开发，推荐在[交互式模式](https://youtu.be/lwN4-W1WR84?t=107)下使用VSCode的[Python扩展](https://marketplace.visualstudio.com/items?itemName=ms-python.python)。 **复现云实验** 本实验方案可自动化开展不同类型负载（恒定负载与突发负载）的基准测试实验，并生成与前文描述格式一致的全新数据集。具体步骤如下： 1. 按照[LOADGENERATOR](./serverless-benchmarker/docs/LOADGENERATOR.md)中的说明，将负载生成器部署为观测节点 2. 在[constant.py](./experiment-plans/constant.py)实验方案中选择目标云厂商`PROVIDER`（aws或azure） 3. 运行[constant.py](./experiment-plans/constant.py)实验方案： - 打开tmux会话 - 激活虚拟环境：执行`source sb-env/bin/activate` - 运行命令：`./constant.py 2>&1 | tee -a constant.log`