The dataset of the APSEC'20 paper titled "Understanding the Non-Repairability Factors of Automated Program Repair Techniques"

The file ExecutionLogs.zip includes all execution logs that we take into consideration in the study. 

The file Exception_Statistics.xlsx includes the distribution of each type of exception in our study. 

All logs are located in directories that are named by fault localization strategy and the corresponding automated program repair tool. Specifically, in the directories of ExecutionLogs there are four sub-directories: GZ-0.1, GZ-1.7, No_FL_Required, and Perfect_FL.

GZ-0.1 means fault localization strategy with GZoltar-v0.1 (from the FSE19 paper).

GZ-1.7 means fault localization strategy with GZoltar-v1.7 (from the ICSE20 paper).

Perfect_FL means directly feeding ground-truth bug positions to APR tools (from the ICSE20 paper).

No_FL_Required means that the tool does not require fault localization process (from FSE19 paper).

For example, the log of bug Chart-1 generated by Cardumen with Gzoltar-0.1 are stored in the directory  GZ-0.1/Chart/1/Cardumen/. 

In the file Exception_Statistics.xlsx, we demonstrate the detailed distribution of each type of exception.

Format of the content

We use format like Fault_localization_strategy-project_id-version_id-tool_name to represent each log.

Take GZ0.1-Chart-1-Cardumen as an example, this log is generated by Cardumen on bug Chart-1 with fault localization strategy GZoltar-v0.1.

About compliance level

Take the log of GZ0.1-Mockito-1-Cardumen as an example. We can get the compliance level from the parameter javacompliancelevel. For example, 4 corresponds to java 1.4, 5 corresponds to java 1.5 and so on. Hence, we can know what compliance level authors used during the original experiment.

time java -Xmx4048m -cp /home/tdurieux/RepairThemAll/script/../libs/jtestex7.jar:/home/tdurieux/RepairThemAll/script/../repair_tools/astor.jar fr.inria.main.evolution.AstorMain \
    -mode cardumen \
    -location . \
    -id Chart-1 \
    -failing org.jfree.chart.renderer.category.junit.AbstractCategoryItemRendererTests \
    -jvm4testexecution /home/tdurieux/openjdk-1.7.0/bin/ \
    -jvm4evosuitetestexecution /home/tdurieux/openjdk-1.7.0/bin/ \
    -maxgen 1000000 \
    -maxtime 120 \
    -stopfirst true \
    -seed 0 \
    -scope local \
    -population 1 \
    -javacompliancelevel 4 \
......

About the server

In the end of each log, it illustrates the brief information about the cloud node. Hence, we can know which logs are from the same node.

......
Node: suno-2.sophia.grid5000.fr

Date: Fri Dec 28 21:21:00 EST 2018

We next give introduction to the omitted three types of exceptions in the paper.

NumberFormat exception

Below is an example of this type of exception. As we have mentioned in the paper, due to the impossibility of integrating GZoltar-v1.7 into APR tools, researchers choose to read the FL results from independent files. From the content of the log, Astor needs to get suspicious line numbers during initialization (executing createEngine) but triggers an exception. This exception only occurs for Closure-76 and we have confirmed with the authors of reference[9] that it is due to the wrong content of the input file. We thus categorized it into Improper operation of replication study.

Exception in thread "main" java.lang.NumberFormatException: For input string: ""
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:592)
at java.lang.Integer.parseInt(Integer.java:615)
at fr.inria.main.evolution.AstorMain.createEngine(AstorMain.java:172)
at ......

IO exception

During the experiment, the generated output files (i.e., the execution logs and the generated patches) can sometimes be more than 10 trillions of bites. Meanwhile, the system may generate and store abundant template files such as the variants of the buggy program. If there is not enough space left on the machine, the execution logs will not be outputted successfully and an IO exception will occur as shown below. This indicates that these exceptions may not occur if the template files can be cleaned in time. We thus classified this type into Improper operation of replication study in that researchers ignore the storage state of the machine.

Exception in thread "main" java.lang.RuntimeException: java.io.IOException: No space left on device
at fr.inria.astor.core.setup.ProjectRepairFacade.setupWorkingDirectories(ProjectRepairFacade.java:46)
at fr.inria.main.evolution.AstorMain.initProject(AstorMain.java:61)
at fr.inria.main.evolution.AstorMain.run(AstorMain.java:239)

NoClassDefFound exception

An example of this type of exception is illustrated below. We failed to reproduce this exception in our own machine. We further tried the docker file provided in the project page of reference[8] but still did not succeed. We noted that the experiment in reference[8] was performed on cloud instances whose experimental environment may be unstable. Bugs that throw this exception are consecutive (i.e., the IDs of these bugs are consecutive which are from Math 39 to Math 52 in the Defects4J benchmark), which further confirms our hypothesis that the environment might be problematic during these executions. We thus categorized this type into Unstable environment.

Exception in thread "main" java.lang.NoClassDefFoundError: com/gzoltar/core/spectra/Spectra
at com.gzoltar.core.GZoltar.run(GZoltar.java:50)
at us.msu.cse.repair.core.faultlocalizer.GZoltarFaultLocalizer.(GZoltarFaultLocalizer.java:42)
at us.msu.cse.repair.core.AbstractRepairProblem.invokeFaultLocalizer(AbstractRepairProblem.java:298)
at us.msu.cse.repair.core.AbstractRepairProblem.invokeModules(AbstractRepairProblem.java:269)
at us.msu.cse.repair.core.AbstractRepairProblem.(AbstractRepairProblem.java:244)
at us.msu.cse.repair.algorithms.kali.Kali.(Kali.java:37)
at ......