Defects4J Workability for APR Results

Results for workability setup-test and trivial patch experiments on Defects4J version 2.0. setup-test-results.csv contains the result of the setup-test algorithm, executed 20 times on the full Defects4J dataset. The first rows contain meta-information on the runs: which of two different hosts for the VM was used, the parallelism setting, the aggregated (counted) results, and the total runtime (wall time) for the complete run. setup-test-without-classpath-fixes.csv contains the same information, but for a single run when not applying the custom fixes for the classpaths as exported by Defects4J. We cannot show our implementation source code here, as that would reveal us as authors. So for the blinded review phase, we provide the following textual description of the classpath fixes (for the final publication, the source code will be made freely available): For the Cli project, the elements file, \${test.classes.dir}, and \${classes.dir} were removed from the test classpath. These are seemingly missing variable expansions. For the Closure project, the libraries lib/junit.jar and lib/json.jar are added to the compilation classpath, and the directory build/classes is added to the compilation and test classpaths. For the Compress project, the element \${dependency.jar} is removed from the test classpath (another missing variable expansion). For the JacksonXml project, the library stax-api-1.0-2.jar is removed from the compilation and test classpaths. For the Mockito project, the library junit-4.11.jar is added to the compilation classpath, the library hamcrest-all-1.3.jar is removed from the compilation and test classpaths, and the directory build/classes/main is added to the test classpath. For the Math project, the library commons-discovery-0.5.jar is added to the compilation classpath of all instances with ID 100 or greater, and the directory src/main/resources is added to the test classpath of all instances up to and including ID 84. Additionally, the class org.apache.commons.math3.genetics.FixedElapsedTimeTest is removed from the test suite for all instances in this project. For the Time project, the generated resources in the package org/joda/time/tz/data are copied over to the source directory, so that they are available for compilation outside of the Defects4J tooling. The file jgenprog-results.csv contains which defects in Defects4J 1.0 were reported to be fixed by jGenProg in the cited paper, as well as whether they are also included in Defects4J 2.0. The file single-statement-deletion-results.csv contains the results of the experiment that deletes single statements to check for trivial fixes. The result of the experiment for each defect is in column 2: "FOUND_FIX" means that the test suite fully passes when deleting a single statement, "IMPROVED" that more test cases pass than originally, but not all, "NO_CHANGE" that no statement deletion can make more test cases pass. Additionally, the experiment was done with a 3 hour timeout for each defect; if that is reached, the result is "TIMEOUT". For all defects where a trivial fix is found ("FOUND_FIX" in column 2), column 3 shows the result of a manual investigation whether the original human-written patch for the defect contains only deletions. Column 4 shows which of the bugs is also in Defects4J version 1.0, and finally column 5 shows which bugs are fixed by jGenProg (see jgenprog-results.csv).