A Road to Find Them All: Towards an Agnostic Strategy for Test Smell Detection

Test smells are indications of potential problems in the design and implementation of automated tests. Unit test automation frame- works (xUnit) generally provide similar features for test setup, veri- fication, and teardown steps. Likewise, test smell detection tools perform similar detection steps for smells across programming lan- guages and xUnit frameworks, which might represent a redundant effort. This paper proposes a unified strategy to detect test smells across xUnit frameworks. To do so, we convert the original test code to a standard format among programming languages and xUnit frameworks and search for smells in it, thus promoting the reuse of test smell detection steps. Also, to demonstrate our strategy in practice, we implement a tool named SniffML, which can detect 7 test smells. As the standard format, SniffML considers the srcML li- brary to convert the test code to XML. To evaluate our strategy and tool, we apply SniffML to analyze 300 from 9 open-source projects written in C++, C#, and Java using GoogleTest, NUnit, and JUnit, respectively. In our study, (i) we evaluate its performance through precision, recall, and f-measure metrics; and (ii) we compare its performance to other popular test smell detection tools (i.e., xNose for C# and JNose and tsDetect for Java). As results, we achieved a precision of 97.99%, a recall of 96.90%, and an f-measure of 97.44%, indicating promising results in detecting test smells across xUnit frameworks. We achieve similar results when comparing SniffML to xNose, JNose and tsDetect, demonstrating the positive potential of our approach. We also contributed to the first test smell detection tool that supports the Google framework.

测试异味（test smells）是自动化测试设计与实现中潜在问题的表征。单元测试自动化框架（xUnit）通常为测试准备、验证与清理步骤提供相似的功能。同理，测试异味检测工具会针对不同编程语言与xUnit框架下的异味执行相似的检测步骤，这可能造成冗余的工作开销。本文提出一种可跨xUnit框架检测测试异味的统一策略。为此，我们将原始测试代码转换为适配多编程语言与xUnit框架的标准格式，并在此格式中搜索异味，以此提升测试异味检测步骤的复用性。此外，为在实践中验证所提策略，我们开发了一款名为SniffML的工具，该工具可检测7种测试异味。在标准格式的实现上，SniffML借助srcML库将测试代码转换为可扩展标记语言（XML）。为评估所提策略与工具的性能，我们将SniffML应用于分析来自9个开源项目的300份测试代码，这些项目分别采用C++、C#与Java语言编写，并分别使用GoogleTest、NUnit与JUnit框架。在本研究中，我们（1）通过精确率（precision）、召回率（recall）与F1度量（f-measure）指标评估工具性能；（2）将本工具的性能与其他主流测试异味检测工具进行对比：针对C#的xNose，以及针对Java的JNose与tsDetect。实验结果显示，本工具的精确率达97.99%、召回率达96.90%、F1度量达97.44%，证明其在跨xUnit框架的测试异味检测任务中具备优异的性能表现。将SniffML与xNose、JNose及tsDetect进行对比后，本工具同样取得了优异的检测结果，进一步验证了所提方法的应用潜力。本研究还开发了首款支持GoogleTest框架的测试异味检测工具。