An empirical study of automatically-generated tests from the perspective of test smells

Developing software test code can be as or more expensive than developing software production code. Commonly, developers use automated unit test generators to speed up software testing. The purpose of such tools is to shorten production time without decreasing code quality. Nonetheless, unit tests usually do not have a quality check layer above testing code, which might be hard to guarantee the quality of the generated tests. An emerging strategy to verify the tests quality is to analyze the presence of test smells in software test code. Test smells are characteristics in the test code that possibly indicate weaknesses in test design and implementation. The presence of test smells in unit test code could be used as an indicator of unit test quality. In this paper, we present an empirical study aimed to analyze the quality of unit test code generated by automated test tools. We compare the tests generated by two tools (Randoop and EvoSuite) with the existing unit test suite of open-source software projects. We analyze the unit test code of twenty-one open-source Java projects and detected the presence of nineteen types of test smells. The results indicated significant differences in the unit test quality when comparing data from both automated unit test generators and existing unit test suites.

开发软件测试代码的成本，往往与开发生产代码持平，甚至更高。通常而言，开发人员会借助自动化单元测试生成工具来加快软件测试流程。此类工具的设计初衷，是在不降低代码质量的前提下缩短研发周期。不过，单元测试通常未在测试代码层级之上设置质量校验环节，因此难以保障生成的测试代码的质量。验证测试质量的一种新兴思路，是分析软件测试代码中是否存在测试异味（test smells）。测试异味指测试代码中可能预示测试设计与实现存在缺陷的特征。单元测试代码中出现的测试异味，可作为衡量单元测试质量的指标。在本研究中，我们实施了一项实证研究，旨在分析自动化测试工具生成的单元测试代码的质量。我们将两款工具（Randoop与EvoSuite）生成的测试代码，与开源软件项目的现有单元测试套件进行对比。我们分析了21个开源Java项目的单元测试代码，共检测出19类测试异味。研究结果显示，对比自动化单元测试生成工具与现有单元测试套件的相关数据，二者的单元测试质量存在显著差异。