Replication Data for "How Closely are Common Mutation Operators Coupled to Real Faults?"

# Replication Data for "How Closely are Common Mutation Operators Coupled to Real Faults?"

## Overview

In mutation testing, faulty versions of a program are generated through automated modifications of source code. These mutants are used to assess and improve test suite quality, under the assumption that detection of mutants is indicative of a test suite's ability to detect real faults - i.e., that mutants and faults have a semantic relationship. Improving the effectiveness - in both cost and quality - of mutation testing may lie in better understanding this relationship, in particular with regard to how individual mutation operators (types) couple to real faults.  

In this study, we examine coupling between 32,002 mutants produced by 31 mutation operators and 144 real faults, using a scale based on number of failing tests and reasons for failure. Ultimately, we observed that 9.92% of the mutants are strongly coupled to real faults, and 51.03% of the faults have at least one strongly coupled mutant. We identify and examine mutation operators with the highest median coupling, as well as the operators that tend to produce non-compiling mutants, undetected mutants, and mutants that cause the most tests to fail outside of the tests that detect the actual fault. We also examine how coupling could be used to filter the set of operators employed, leading to potentially significant cost savings during mutation testing. Our findings could lead to improvements in how mutation testing is applied, improved implementation of specific mutation operators, and inspiration for new mutation operators. 

## Data Contained in This Package

- mutant_data.csv

This dataset contains the coupling results for all mutants considered in our experiments. It contains the following attributes for each mutant:

-- Project name from Defects4J
-- Fault number from Defects4J
-- Mutation ID
-- Mutation operator
-- Number of trigger tests for the fault (tests that detect the real fault)
-- Number of failing test cases for the mutant (-1 indicates a compilation error)
-- The number of failing trigger tests for the mutant
-- The number of trigger tests that fail for the same reason the tests failed for the real fault.
-- The number of failing non-trigger tests.
-- The categorization of coupling. In order: Compile Error, Not Detected, No Substitution, Partial Test Substitution + Additional Tests Fail, Partial Test Substitution, Partial Substitution + Additional Tests Fail, Partial Substitution, Test Substitution + Additional Tests Fail, Test Substitution, Strong Substitution + Additional Tests Fail, Strong Substitution. 

- mutant_logs/{Project}/{Project}{Fault Number}output.txt

The raw output log that resulted from executing test cases for each mutant for each case example used from Defects4J. Used to generate the dataset discussed above. Scripting for generating the dataset is also included.