A Road to Find Them All: Towards an Agnostic Strategy for Test Smell Detection

Test smells are indications of potential problems in the design and implementation of automated tests. Unit test automation frame- works (xUnit) generally provide similar features for test setup, veri- fication, and teardown steps. Likewise, test smell detection tools perform similar detection steps for smells across programming lan- guages and xUnit frameworks, which might represent a redundant effort. This paper proposes a unified strategy to detect test smells across xUnit frameworks. To do so, we convert the original test code to a standard format among programming languages and xUnit frameworks and search for smells in it, thus promoting the reuse of test smell detection steps. Also, to demonstrate our strategy in practice, we implement a tool named SniffML, which can detect 7 test smells. As the standard format, SniffML considers the srcML li- brary to convert the test code to XML. To evaluate our strategy and tool, we apply SniffML to analyze 300 from 9 open-source projects written in C++, C#, and Java using GoogleTest, NUnit, and JUnit, respectively. In our study, (i) we evaluate its performance through precision, recall, and f-measure metrics; and (ii) we compare its performance to other popular test smell detection tools (i.e., xNose for C# and JNose and tsDetect for Java). As results, we achieved a precision of 97.99%, a recall of 96.90%, and an f-measure of 97.44%, indicating promising results in detecting test smells across xUnit frameworks. We achieve similar results when comparing SniffML to xNose, JNose and tsDetect, demonstrating the positive potential of our approach. We also contributed to the first test smell detection tool that supports the Google framework.