The dataset of the APSEC'20 paper titled "Understanding the Non-Repairability Factors of Automated Program Repair Techniques"

The file <strong>ExecutionLogs.zip</strong> includes all execution logs that we take into consideration in the study. The file <strong>Exception_Statistics.xlsx</strong> includes the distribution of each type of exception in our study. All logs are located in directories that are named by fault localization strategy and the corresponding automated program repair tool. Specifically, in the directories of <strong>ExecutionLogs</strong> there are four sub-directories: <em>GZ-0.1</em>, <em>GZ-1.7</em>, <em>No_FL_Required</em>, and <em>Perfect_FL</em>. <em>GZ-0.1</em> means fault localization strategy with GZoltar-v0.1 (from the FSE19 paper). <em>GZ-1.7</em> means fault localization strategy with GZoltar-v1.7 (from the ICSE20 paper). <em>Perfect_FL</em> means directly feeding ground-truth bug positions to APR tools (from the ICSE20 paper). <em>No_FL_Required</em> means that the tool does not require fault localization process (from FSE19 paper). For example, the log of bug <em>Chart-1</em> generated by <em>Cardumen</em> with <em>Gzoltar-0.1</em> are stored in the directory <strong>GZ-0.1/Chart/1/Cardumen/</strong>. In the file <strong>Exception_Statistics.xlsx</strong>, we demonstrate the detailed distribution of each type of exception. <strong>Format of the content</strong> We use format like <em>Fault_localization_strategy-project_id-version_id-tool_name</em> to represent each log. Take <em>GZ0.1-Chart-1-Cardumen</em> as an example, this log is generated by <em>Cardumen</em> on bug <em>Chart-1</em> with fault localization strategy <em>GZoltar-v0.1</em>. <strong>About compliance level</strong> Take the log of <em>GZ0.1-Mockito-1-Cardumen</em> as an example. We can get the compliance level from the parameter <em>javacompliancelevel</em>. For example, 4 corresponds to java 1.4, 5 corresponds to java 1.5 and so on. Hence, we can know what compliance level authors used during the original experiment. <pre><code class="language-java">time java -Xmx4048m -cp /home/tdurieux/RepairThemAll/script/../libs/jtestex7.jar:/home/tdurieux/RepairThemAll/script/../repair_tools/astor.jar fr.inria.main.evolution.AstorMain \ -mode cardumen \ -location . \ -id Chart-1 \ -failing org.jfree.chart.renderer.category.junit.AbstractCategoryItemRendererTests \ -jvm4testexecution /home/tdurieux/openjdk-1.7.0/bin/ \ -jvm4evosuitetestexecution /home/tdurieux/openjdk-1.7.0/bin/ \ -maxgen 1000000 \ -maxtime 120 \ -stopfirst true \ -seed 0 \ -scope local \ -population 1 \ -javacompliancelevel 4 \ ......</code></pre> <strong>About the server</strong> In the end of each log, it illustrates the brief information about the cloud node. Hence, we can know which logs are from the same node. <pre><code class="language-java">...... Node: suno-2.sophia.grid5000.fr Date: Fri Dec 28 21:21:00 EST 2018</code></pre> We next give introduction to the omitted three types of exceptions in the paper. <strong>NumberFormat exception</strong> Below is an example of this type of exception. As we have mentioned in the paper, due to the impossibility of integrating GZoltar-v1.7 into APR tools, researchers choose to read the FL results from independent files. From the content of the log, Astor needs to get suspicious line numbers during initialization (executing <em>createEngine</em>) but triggers an exception. This exception only occurs for Closure-76 and we have confirmed with the authors of reference[9] that it is due to the wrong content of the input file. We thus categorized it into <strong>Improper operation of replication study</strong>. <pre><code class="language-java">Exception in thread "main" java.lang.NumberFormatException: For input string: "" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Integer.parseInt(Integer.java:592) at java.lang.Integer.parseInt(Integer.java:615) at fr.inria.main.evolution.AstorMain.createEngine(AstorMain.java:172) at ......</code></pre> <strong>IO exception</strong> During the experiment, the generated output files (i.e., the execution logs and the generated patches) can sometimes be more than 10 trillions of bites. Meanwhile, the system may generate and store abundant template files such as the variants of the buggy program. If there is not enough space left on the machine, the execution logs will not be outputted successfully and an IO exception will occur as shown below. This indicates that these exceptions may not occur if the template files can be cleaned in time. We thus classified this type into <strong>Improper operation of replication study</strong> in that researchers ignore the storage state of the machine. <pre><code class="language-java">Exception in thread "main" java.lang.RuntimeException: java.io.IOException: No space left on device at fr.inria.astor.core.setup.ProjectRepairFacade.setupWorkingDirectories(ProjectRepairFacade.java:46) at fr.inria.main.evolution.AstorMain.initProject(AstorMain.java:61) at fr.inria.main.evolution.AstorMain.run(AstorMain.java:239)</code></pre> <strong>NoClassDefFound exception</strong> An example of this type of exception is illustrated below. We failed to reproduce this exception in our own machine. We further tried the docker file provided in the project page of reference[8] but still did not succeed. We noted that the experiment in reference[8] was performed on cloud instances whose experimental environment may be unstable. Bugs that throw this exception are consecutive (i.e., the IDs of these bugs are consecutive which are from Math 39 to Math 52 in the Defects4J benchmark), which further confirms our hypothesis that the environment might be problematic during these executions. We thus categorized this type into <strong>Unstable environment</strong>. <pre><code class="language-java">Exception in thread "main" java.lang.NoClassDefFoundError: com/gzoltar/core/spectra/Spectra at com.gzoltar.core.GZoltar.run(GZoltar.java:50) at us.msu.cse.repair.core.faultlocalizer.GZoltarFaultLocalizer.&lt;init&gt;(GZoltarFaultLocalizer.java:42) at us.msu.cse.repair.core.AbstractRepairProblem.invokeFaultLocalizer(AbstractRepairProblem.java:298) at us.msu.cse.repair.core.AbstractRepairProblem.invokeModules(AbstractRepairProblem.java:269) at us.msu.cse.repair.core.AbstractRepairProblem.&lt;init&gt;(AbstractRepairProblem.java:244) at us.msu.cse.repair.algorithms.kali.Kali.&lt;init&gt;(Kali.java:37) at ......</code></pre>