Defects4J Reproducibility Experiment Results

Author information is hidden in this submission for double-blinded review process. setup-test-run-descriptions.csv contains information about the executions of the setup-test algorithm on all 835 Defects4J defects. 10 runs were executed. The parallelism column describes how many defects were analyzed in parallel. The coverage matrix describes a technical detail, how the test-cases were individually executed with coverage information (full means coverage is taken for each single test-case, while simple means that this is done only for test classes were at least one test fails; for all completely passing test classes, only the coverage for all tests it collected once aggregated). The experiments were conducted on two different host machines (though the VM setup is identical); host A is faster than host B. The runtime is simply the wall-time from start to finish. setup-test-results.csv contains the results of the setup-test algorithm for the 835 defects for all runs. For convenience, the workable column contains whether all 10 executions passed, while the unstable column contains whether the first execution passed and any of the following fails. The result for a single run on a single bug are: MATCHING_DEFECTS4J means that the test passed, ORIGINAL_UNFIT means that compilation failed or the test-suite is inconsistent, and NOT_MATCHING_DEFECTS4J means that the list of failing test cases differs from what is expected in the Defects4J dataset. single-statement-deletion-results.csv contains the result of the second experiment. It lists for all 658 workable bugs, whether a fix was found by deleting a single one of the candidate statements.