Instance Space Analysis of Search-Based Software Testing

Search-based software testing (SBST) is now a mature area, with numerous techniques developed to tackle the challenging task of software testing. SBST techniques have shown promising results and have been successfully applied in the industry to automatically generate test cases for large and complex software systems. Their effectiveness, however, has been shown to be problem dependent. In this paper, we revisit the problem of objective performance evaluation of SBST techniques in light of recent methodological advances – in the form of Instance Space Analysis (ISA) – enabling the strengths and weaknesses of SBST techniques to be visualised and assessed across the broadest possible space of problem instances (software classes) from common benchmark datasets. We identify features of SBST problems that explain why a particular instance is hard for an SBST technique, reveal areas of hard and easy problems in the instance space of existing benchmark datasets, and identify the strengths and weaknesses of state-of-the-art SBST techniques. In addition, we examine the diversity and quality of common benchmark datasets used in experimental evaluations.

基于搜索的软件测试（SBST）领域现已成熟，众多技术已被开发出来以应对软件测试这一挑战性的任务。SBST技术展现出令人鼓舞的结果，并在业界成功应用于自动生成大型和复杂软件系统的测试用例。然而，其有效性已被证明是依赖于特定问题的。本文在近年来方法论上的进步——以实例空间分析（ISA）的形式——的背景下，重新审视了SBST技术客观性能评估的问题。ISA方法使得SBST技术的优势和劣势能够在尽可能广泛的实例空间（软件类别）中得到可视化评估，这一空间涵盖了来自常见基准数据集的广泛问题实例。我们确定了SBST问题的特征，解释了为何某个特定实例对SBST技术来说很难处理，揭示了现有基准数据集实例空间中难题和易题的领域，并确定了最先进SBST技术的优势和劣势。此外，我们还考察了实验评估中使用的常见基准数据集的多样性和质量。