The State of Serverless Applications: Collection,Characterization, and Community Consensus - Replication Package
收藏资源简介：
The replication package for our article <em>The State of Serverless Applications: Collection,Characterization, and Community Consensus</em> provides everything required to reproduce all results for the following three studies: Serverless Application Collection Serverless Application Characterization Comparison Study <strong>Serverless Application Collection</strong> We collect descriptions of serverless applications from open-source projects, academic literature, industrial literature, and scientific computing. <em>Open-source Applications</em> As a starting point, we used an existing data set on open-source serverless projects from this study. We removed small and inactive projects based on the number of files, commits, contributors, and watchers. Next, we manually filtered the resulting data set to include only projects that implement serverless applications. We provide a table containing all projects that remained after the filtering alongside the notes from the manual filtering. <em>Academic Literature Applications</em> We based our search on an existing community-curated dataset on literature for serverless computing consisting of over 180 peer-reviewed articles. First, we filtered the articles based on title and abstract. In a second iteration, we filtered out any articles that implement only a single function for evaluation purposes or do not include sufficient detail to enable a review. As the authors were familiar with some additional publications describing serverless applications, we contributed them to the community-curated dataset and included them in this study. We provide a table with our notes from the manual filtering. <em>Scientific Computing Applications</em> Most of these scientific computing serverless applications are still at an early stage and therefore there is little public data available. One of the authors is employed at the German Aerospace Center (DLR) at the time of writing, which allowed us to collect information about several projects at DLR that are either currently moving to serverless solutions or are planning to do so. Additionally, an application from the German Electron Synchrotron (DESY) could be included. For each of these scientific computing applications, we provide a document containing a description of the project and the names of our contacts that provided information for the characterization of these applications. SC1 Copernicus Sentinel-1 for near-real-time water monitoring SC2 Reprocessing Sentinel 5 Precursor data with ProEO SC3 High-Performance Data Analytics for Earth Observation SC4 Tandem-L exploitation platform SC5 Global Urban Footprint SC6 DESY - High Throughput Data Taking <em>Collection of serverless applications</em> Based on the previously described methodology, we collected a diverse dataset of 89 serverless applications from open-source projects, academic literature, industrial literature, and scientific computing. This dataset is can be found in Dataset.xlsx. <strong>Serverless Application Characterization</strong> As previously described, we collected 89 serverless applications from four different sources. Subsequently, two randomly assigned reviewers out of seven available reviewers characterized each application along 22 characteristics in a structured collaborative review sheet. The characteristics and potential values were defined a priori by the authors and iteratively refined, extended, and generalized during the review process. The initial moderate inter-rater agreement was followed by a discussion and consolidation phase, where all differences between the two reviewers were discussed and resolved. The six scientific applications were not publicly available and therefore characterized by a single domain expert, who is either involved in the development of the applications or in direct contact with the development team. <em>Initial Ratings & Interrater Agreement Calculation</em> The initial reviews are available as a table, where every application is characterized along with the 22 characteristics. A single value indicates that both reviewers assigned the same value, whereas a value of the form <code>[Reviewer 2] A | [Reviewer 4] B</code> indicates that for this characteristic, reviewer two assigned the value A, whereas reviewer assigned the value B. Our script for the calculation of the Fleiß-Kappa score based on this data is also publically available. It requires the python package <code>pandas</code> and <code>statsmodels</code>. It does not require any input and assumes that the file <code>Initial Characterizations.csv</code> is located in the same folder. It can be executed as follows: <pre><code>python3 CalculateKappa.py </code></pre> <em>Results Including Unknown Data</em> In the following discussion and consolidation phase, the reviewers compared their notes and tried to reach a consensus for the characteristics with conflicting assignments. In a few cases, the two reviewers had different interpretations of a characteristic. These conflicts were discussed among all authors to ensure that characteristic interpretations were consistent. However, for most conflicts, the consolidation was a quick process as the most frequent type of conflict was that one reviewer found additional documentation that the other reviewer did not find. For six characteristics, many applications were assigned the ''Unknown'' value, i.e., the reviewers were not able to determine the value of this characteristic. Therefore, we excluded these characteristics from this study. For the remaining characteristics, the percentage of ''Unknowns'' ranges from 0–19% with two outliers at 25% and 30%. These ''Unknowns'' were excluded from the percentage values presented in the article. As part of our replication package, we provide the raw results for each characteristic including the ''Unknown'' percentages in the form of bar charts. The script for the generation of these bar charts is also part of this replication package). It uses the python packages <code>pandas</code>, <code>numpy</code>, and <code>matplotlib</code>. It does not require any input and assumes that the file <code>Dataset.csv</code> is located in the same folder. It can be executed as follows: <pre><code>python3 GenerateResultsIncludingUnknown.py </code></pre> <em>Final Dataset & Figure Generation</em> In the following discussion and consolidation phase, the reviewers compared their notes and tried to reach a consensus for the characteristics with conflicting assignments. In a few cases, the two reviewers had different interpretations of a characteristic. These conflicts were discussed among all authors to ensure that characteristic interpretations were consistent. However, for most conflicts, the consolidation was a quick process as the most frequent type of conflict was that one reviewer found additional documentation that the other reviewer did not find. Following this process, we were able to resolve all conflicts, resulting in a collection of 89 applications described by 18 characteristics. This dataset is available here: link The script to generate all figures shown in the chapter "Serverless Application Characterization can be found here. It does not require any input but assumes that the file <code>Dataset.csv</code> is located in the same folder. It uses the python packages <code>pandas</code>, <code>numpy</code>, and <code>matplotlib</code>. It can be executed as follows: <pre><code>python3 GenerateFigures.py </code></pre> <em>Comparison Study</em> To identify existing surveys and datasets that also investigate one of our characteristics, we conducted a literature search using Google as our search engine, as we were mostly looking for grey literature. We used the following search term: <pre><code>("serverless" OR "faas") AND ("dataset" OR "survey" OR "report") after: 2018-01-01 </code></pre> This search term looks for any combination of either serverless or faas alongside any of the terms dataset, survey, or report. We further limited the search to any articles after 2017, as serverless is a fast-moving field and therefore any older studies are likely outdated already. This search term resulted in a total of 173 search results. In order to validate if using only a single search engine is sufficient, and if the search term is broad enough, we checked if the seven studies the authors were already familiar with are contained in the search results. As all seven studies were contained in the search results, we concluded that the literature search was broad enough. In a first iteration, we filtered out all results that do not either report original data or report on data from another study. Next, we removed all reports on secondary data, where the original study was already contained in the search results. This process resulted in a total of 16 identified studies. Finally, we determined for each identified study if they investigate one of our characteristics. This resulted in a total of ten related studies. The results from the literature search and the notes from the filtering are part of this replication package as a table. As these studies use different answer options than our study, we mapped their answer options to ours. In many cases, this was straightforward, such as mapping HTTP to HTTP Request. If the answer option granularities between the studies differed, we aggregated answer options from the study with lower granularity to match the higher granularity study. In case the lower granularity study allowed multiple answers, we selected only the highest value instead of aggregating them, to avoid counting a single study participant multiple times. As this mapping process is somewhat subjective, we provide a detailed account of the mapping for each characteristic and related study as a multi-sheet excel table, where each sheet shows our mapping alongside our notes which answer options were mapped. For many studies, not all information required for traditional meta-analysis techniques, such as cohort size, is available, preventing the application of these meta-analysis techniques. Therefore, we came up with an agreement metric that equally weights the agreement of the reported ranking and the agreement of the reported percentage values. It combines the relative difference between the reported percentages of both studies and the order of the reported popularities of the answer options. We categorize scores in the range [0.8, 1] as very high agreement, [0.6,0.6[ as high agreement, [0.4, 0.6[ as medium agreement, [0.2, 0.4[ as low agreement, and [0, 0.2[ as very low agreement. We acknowledge that these categories are somewhat arbitrary, however, based on a manual inspection of the results, they do seem to capture the level of agreement between the individual studies quite well. Our replication package includes the mapped data alongside the resulting scores to enable a manual inspection of the degree of agreement. The script that implements the calculation of our score is also publically available. It uses the python packages <code>pandas</code>, <code>numpy</code>, and <code>scipy</code>. The script does not require any input and assumes that the file <code>Comparison Mappings.xlsx</code> is located in the same folder. The script can be executed as follows: <pre><code>python3 corroboration_analysis.py </code></pre> Further, the script to generate all figures shown in the chapter <em>Comparison Analysis</em> is the final piece of our replication package: link. It uses the python packages <code>numpy</code> and <code>matplotlib</code>. It does not require any input and can be executed as follows: <pre><code>python3 barcharts.py </code></pre> If you have any questions about our study or require any additional information/data please contact the first author.
本研究论文《The State of Serverless Applications: Collection, Characterization, and Community Consensus》的复现包提供了复现以下三项研究全部结果所需的全部材料：无服务器应用（Serverless Application）收集、无服务器应用特征分析（Serverless Application Characterization）与对比研究（Comparison Study）。 ### 无服务器应用收集（Serverless Application Collection） 我们从开源项目、学术文献、工业文献以及科学计算场景中收集无服务器应用的描述。 #### 开源应用（Open-source Applications） 我们以本研究已有的开源无服务器项目数据集作为起点，基于文件数量、提交记录、贡献者与关注者数量，剔除小型且不活跃的项目。随后手动筛选所得数据集，仅保留实现无服务器应用的项目。我们提供了一份表格，包含筛选后剩余的全部项目，以及手动筛选过程中的备注。 #### 学术文献应用（Academic Literature Applications） 我们的检索基于现有由社区整理的、包含180余篇同行评议论文的无服务器计算文献数据集。首先根据标题与摘要筛选论文；第二轮筛选则剔除仅实现单个函数用于评估，或缺乏足够细节无法开展分析的论文。由于作者团队熟悉部分额外的无服务器应用相关出版物，我们将其补充至该社区整理数据集，并纳入本研究。我们提供了一份表格，包含手动筛选过程中的备注。 #### 科学计算应用（Scientific Computing Applications） 此类科学计算无服务器应用大多仍处于早期阶段，公开可用数据极少。本文撰写时，其中一位作者任职于德国航空航天中心（German Aerospace Center, DLR），借此我们收集了该中心内多个正转向或计划转向无服务器解决方案的项目信息。此外，还纳入了德国电子同步加速器（German Electron Synchrotron, DESY）的一款应用。针对每一项科学计算应用，我们提供了一份文档，包含项目描述以及为这些应用的特征分析提供信息的联系人姓名。具体应用包括： SC1 用于近实时水体监测的哥白尼哨兵-1号（Copernicus Sentinel-1） SC2 使用ProEO处理哨兵-5号 precursor 数据 SC3 面向地球观测的高性能数据分析 SC4 Tandem-L 开发平台 SC5 全球城市足迹（Global Urban Footprint） SC6 DESY 高通量数据采集 基于前述方法，我们从四类来源中收集了共89款多样化的无服务器应用数据集，该数据集可在Dataset.xlsx中获取。 ### 无服务器应用特征分析（Serverless Application Characterization） 我们从四类来源收集了89款无服务器应用。随后，从7名评审中随机指派2名评审，通过结构化协作评审表，针对每款应用的22项特征开展特征分析。特征项与可选值由作者团队预先定义，并在评审过程中迭代优化、扩展与泛化。初始的评审者间一致性评分完成后，进入讨论与整合阶段，评审双方针对所有分歧展开讨论并达成统一。6款未公开的科学计算应用，则由一名参与应用开发或直接对接开发团队的领域专家单独完成特征分析。 #### 初始评分与评审者间一致性计算（Initial Ratings & Interrater Agreement Calculation） 初始评审结果以表格形式提供，其中每款应用的22项特征均有记录。单一数值表示两名评审给出了相同的特征值，而形如`[Reviewer 2] A | [Reviewer 4] B`的数值表示该特征项中，评审2赋值为A，评审4赋值为B。我们基于该数据计算弗莱施卡帕系数（Fleiss-Kappa）的脚本也已公开。该脚本依赖Python包`pandas`与`statsmodels`，无需额外输入，仅需将`Initial Characterizations.csv`文件置于同一目录下。执行方式如下： <pre><code>python3 CalculateKappa.py</code></pre> #### 包含未知数据的分析结果（Results Including Unknown Data） 在后续的讨论与整合阶段，评审双方比对各自的记录，尝试就存在分歧的特征项达成共识。少数情况下，两名评审对某一特征项存在不同解读，此类分歧会提交全体作者讨论，以确保特征项解读的一致性。不过多数分歧的解决过程较为高效，最常见的分歧类型为其中一名评审找到了另一名评审未发现的额外文档。针对6项特征，多数应用被赋值为“未知”，即评审无法确定该特征的取值，因此我们将这6项特征从本研究中剔除。剩余特征项的“未知”占比介于0~19%，仅两项异常值分别为25%与30%。文章中呈现的百分比数据已剔除“未知”项。作为复现包的一部分，我们提供了每项特征的原始结果，包括“未知”占比的柱状图。生成此类柱状图的脚本同样包含在复现包中，该脚本依赖Python包`pandas`、`numpy`与`matplotlib`，无需额外输入，仅需将`Dataset.csv`文件置于同一目录下。执行方式如下： <pre><code>python3 GenerateResultsIncludingUnknown.py</code></pre> #### 最终数据集与图表生成（Final Dataset & Figure Generation） 在讨论与整合阶段，评审双方比对记录并就分歧特征项达成共识。少数情况下，评审对特征项存在不同解读，此类分歧提交全体作者讨论以统一解读。多数分歧的解决过程高效，最常见的分歧为其中一名评审找到了另一名未发现的额外文档。经此流程，我们解决了所有分歧，最终得到由18项特征描述的89款应用的数据集。该数据集可在此处获取：[链接]。生成“无服务器应用特征分析”章节中所有图表的脚本可在此处获取。该脚本无需额外输入，仅需将`Dataset.csv`文件置于同一目录下，依赖Python包`pandas`、`numpy`与`matplotlib`。执行方式如下： <pre><code>python3 GenerateFigures.py</code></pre> ### 对比研究（Comparison Study） 为识别同样研究了我们所关注特征项的现有综述与数据集，我们以谷歌（Google）作为搜索引擎开展文献检索，主要目标为灰色文献。使用的检索式如下： <pre><code>("serverless" OR "faas") AND ("dataset" OR "survey" OR "report") after: 2018-01-01</code></pre> 该检索式匹配包含“serverless”或“faas”，同时包含“dataset”“survey”或“report”的文献。我们进一步将检索范围限定在2017年之后的文献，由于无服务器领域发展迅速，较早的研究大概率已过时。本次检索共得到173条结果。为验证仅使用单一搜索引擎是否足够，且检索式是否覆盖足够广泛，我们检查了作者团队已熟悉的7项研究是否均包含在检索结果中，结果全部包含在内，因此我们认为本次文献检索的覆盖范围足够。 第一轮筛选：剔除所有未报告原始数据或仅转述其他研究数据的结果。第二轮筛选：移除所有仅报告二次数据且其原始研究已包含在检索结果中的报告。此流程共得到16项已识别的研究。最后，我们逐一判定每一项已识别的研究是否涉及我们所关注的特征项，最终得到10项相关研究。文献检索结果与筛选过程的备注以表格形式包含在本复现包中。 由于这些研究使用的答案选项与本研究不同，我们将其答案选项映射至本研究的选项体系。多数情况下映射较为直接，例如将HTTP映射为HTTP请求。若研究间的答案选项粒度存在差异，我们将低粒度研究的选项聚合以匹配高粒度研究的选项。若低粒度研究允许多选，我们仅选择最高优先级的选项，以避免重复计数单个研究参与者。由于该映射过程存在一定主观性，我们针对每项特征与相关研究的映射过程提供了一份多工作表Excel表格，每个工作表展示了我们的映射方案以及所映射的答案选项备注。 多数研究无法提供传统元分析技术所需的全部信息（例如队列规模），因此无法使用此类元分析方法。为此，我们提出了一种一致性度量指标，对报告的排名一致性与报告的百分比值一致性赋予同等权重。该指标结合了两项研究报告的百分比之间的相对差异，以及答案选项报告的流行度顺序。我们将得分区间[0.8, 1]划分为极高一致性，[0.6, 0.8]为高一致性，[0.4, 0.6]为中等一致性，[0.2, 0.4]为低一致性，[0, 0.2]为极低一致性。我们承认此类分类存在一定主观性，但基于对结果的手动检视，该分类能够较好地反映各项研究间的一致性程度。本复现包包含映射后的数据与最终得分，以便手动检视一致性程度。实现该指标计算的脚本也已公开，该脚本依赖Python包`pandas`、`numpy`与`scipy`，无需额外输入，仅需将`Comparison Mappings.xlsx`文件置于同一目录下。执行方式如下： <pre><code>python3 corroboration_analysis.py</code></pre> 此外，生成“对比分析”章节中所有图表的脚本是本复现包的最后一部分：[链接]。该脚本依赖Python包`numpy`与`matplotlib`，无需额外输入，执行方式如下： <pre><code>python3 barcharts.py</code></pre> 若您对本研究有任何疑问，或需要额外的信息与数据，请联系第一作者。
