Flaky Test Dataset to Accompany "FlakeFlagger: Predicting Flakiness Without Rerunning Tests"

When developers make changes to their code, they typically run regression tests to detect if their recent changes (re)introduce any bugs. However, many tests are flaky, and their outcomes can change non-deterministically, failing without apparent cause. Flaky tests are a significant nuisance in the development process, since they make it more difficult for developers to trust the outcome of their tests. The traditional approach to identify flaky tests is to rerun them multiple times: if a test is observed both passing and failing on the same code, it is definitely flaky. We conducted a very large empirical study looking for flaky tests by rerunning the test suites of 24 projects 10,000 times each, and found that even with this many reruns, some flaky tests were still not detected. We propose FlakeFlagger, a novel approach that collects a set of features describing the behavior of each test, and then predicts tests that are likely to be flaky based on similar behavioral features. We found that FlakeFlagger correctly labeled at least as many tests as flaky as a state-of-the-art flaky test classifier, but that FlakeFlagger reported far fewer false positives (an increase in precision from just 11% to 60%). This lower false positive rate translates directly to saved time for researchers and developers who use the classification result to guide more expensive flaky test detection processes. By investigating the information gain of each feature, we conclude that test execution time, overall test coverage, coverage of recently changed lines and usage of third party libraries are effective predictors of test flakiness. We did not find any keywords or tokens in the source code of tests that were effective in predicting test flakiness, and did not find the presence of test smells to be effective in predicting test flakiness. This archive contains the dataset that we collected of flaky tests, along with the features that we collected from each test. Contents:<br> Project_Info.csv: List of projects and their revisions studied<br> build-logs-&lt;project-slug&gt;.tgz: An archive of all of the maven build logs from each of the 10,000 runs of that project's test suite. <br> failing-test-reports-&lt;project-slug&gt;.tgz An archive of all of the surefire XML reports for each failing test of each build of each project.<br> test_results.csv: Summary of the number of passing and failing runs for each test in each project. <br> "Run ID" is a key into the &lt;project-slug&gt;.tgz archive also in this artifact, which refers to the run that we observed the test fail on.<br> test_features.csv: Summary of the features that each test had, as per our feature detectors described in the paper<br> flakeflagger-code.zip: All scripts used to generate and process these results. These scripts are also located at https://github.com/AlshammariA/FlakeFlagger开发人员对代码进行修改时，通常会运行回归测试以检测近期的代码变更是否重新引入了漏洞。然而，大量测试属于不稳定测试（flaky test），其执行结果会以非确定方式发生变化，在无明显诱因的情况下出现失败。不稳定测试会给开发流程带来极大困扰，因为它们会降低开发人员对测试结果的信任度。传统的不稳定测试识别方法是多次重复运行测试：若同一测试在相同代码环境下同时出现通过与失败的结果，则可确定其为不稳定测试。 本研究开展了一项大规模实证研究：对24个项目的测试套件（test suite）各执行10000次重复运行以寻找不稳定测试，结果发现即便进行了如此多的重复运行，仍有部分不稳定测试未能被检测出来。 为此我们提出了FlakeFlagger这一全新方法：该方法会收集描述每个测试行为的特征集合，并基于相似的行为特征预测存在不稳定风险的测试。研究结果显示，FlakeFlagger能够正确标记的不稳定测试数量至少与当前主流的不稳定测试分类器相当，但其误报率要低得多——精确率从仅11%提升至60%。更低的误报率可直接为研究人员与开发人员节省时间，这些开发者会借助该分类结果来指导成本更高的不稳定测试检测流程。 通过分析每个特征的信息增益，我们得出结论：测试执行时间、整体测试覆盖率（test coverage）、近期修改代码行的覆盖率以及第三方库的使用情况，是预测测试不稳定性的有效指标。我们未在测试源代码中发现可有效预测测试不稳定性的关键词或Token，同时也未发现测试坏味（test smell）与测试不稳定性存在关联。 本归档文件包含我们收集的不稳定测试数据集，以及从每个测试中提取的特征信息。归档内容如下： Project_Info.csv: 本次研究涉及的项目及其修订版本列表 build-logs-<project-slug>.tgz: 该项目测试套件10000次运行所生成的全部Maven构建日志归档文件 failing-test-reports-<project-slug>.tgz: 所有项目每次构建中所有失败测试的Surefire XML报告归档文件 test_results.csv: 每个项目中各测试的通过与失败运行次数汇总 "Run ID" 是本归档文件中<project-slug>.tgz归档的索引键，用于指向本次观察到测试失败的运行记录 test_features.csv: 依据本文所述特征检测器提取的各测试特征汇总文件 flakeflagger-code.zip: 用于生成与处理本次研究结果的全部脚本文件，这些脚本也可在https://github.com/AlshammariA/FlakeFlagger获取