<b>TaRBench: A Comprehensive Benchmark for Automated Test Case Repair</b>

<b>Abstract</b>Ensuring the quality of software systems through testing is a critical aspect of software development. However, the maintenance of test cases presents significant challenges, both in terms of complexity and cost. The constant need for updates to align with evolving systems under test can result in broken test cases, leading to a deterioration in test suite quality and disruptions in the software development process. To address these challenges, we introduce TaRGet (Test Repair GEneraTor), an approach that leverages pre-trained code language models for automated test case repair. TaRGet treats test repair as a language translation task and employs a two-step process to fine-tune a language model using essential context data that characterizes test breakages.<b>Publication</b>This repository is a supplement to our paper (accepted at IEEE TSE) which can be found on TSE.2025.3541166. For detailed definitions, experiments, and results, please refer to the paper. If you find this repository useful, kindly cite our work:<br><pre>@ARTICLE{saboor2025target,<br> author={Saboor Yaraghi, Ahmadreza and Holden, Darren and Kahani, Nafiseh and Briand, Lionel},<br> journal={IEEE Transactions on Software Engineering}, <br> title={Automated Test Case Repair Using Language Models}, <br> year={2025},<br> volume={51},<br> number={4},<br> pages={1104-1133},<br> doi={10.1109/TSE.2025.3541166}<br>}</pre><b>TaRBench</b>TaRBench is a comprehensive benchmark that we developed to evaluate the effectiveness of TaRGet in automated test case repair. The benchmark encompasses 45,373 broken test repairs across 59 open-source projects, providing a diverse and extensive dataset for assessing the capabilities of TaRGet. In the following sections, we present detailed information about TaRBench (available in the <i>"TaRBench.zip"</i> file).<b>The Data</b>Each project's data is stored in the <i>"projects"</i> directory, following the format <i>"projects/GitHub_Username/GitHub_Repo_Name"</i> (e.g., <i>"projects/apache/druid"</i>). TaRBench structures the data for each project into four JSON files:<b>dataset.json: </b>The main file containing all the test repair instances of the project.<b>codeMining/call_graphs.json: </b>Call graphs of the test cases found in dataset.json across different commits.<b>codeMining/sut_method_changes.json: </b>Code changes in methods of the System Under Test (SUT) commits.<b>codeMining/sut_class_changes.json: </b>Code changes in classes of the SUT commits.Furthermore, the <b>splits.csv</b> file specifies the data split (train, valid, or test) for each test repair sample, identified by its unique ID.In following, we provide details on the attributes within each file.<b>dataset.json</b>The <i>dataset.json</i> file comprises an array of JSON objects, each representing a test repair instance. These instances have the following attributes:<b>ID </b>(String): A unique identifier for the repair instance.<b>name </b>(String): The qualified name of the test case method, including the package, class, and method names.<b>bCommit, aCommit </b>(String): The commit hash (version) of the project before (bCommit) and after (aCommit) the repair.<b>aCommitTime </b>(Integer): Timestamp of aCommit.<b>bPath, aPath </b>(String): The relative path of the test case source file in the project before (bPath) and after (aPath) the repair.<b>bSource</b>, <b>aSource </b>(Object): The source code of the test case method before (bSource) and after (aSource) the repair. Each includes the start line of the test method in its source file as well as the test method source code.<b>hunk </b>(Object): The Git hunk of the test repair, representing the changes made to the test case method for repair. It includes the lines changed before (sourceChanges) and after (targetChanges) the repair, along with the corresponding code elements.<b>verdict </b>(Object): The result of executing the broken test case on the updated version of the SUT, which indicates the type of test failure. It has a `status` which can be either "compile_error" or "failure" (runtime error).<b>astActions </b>(Array): The edit actions applied to the abstract syntax tree (AST) of the test code for repair.<b>trivial </b>(Array): Indicates whether the repair is trivial (class or method renaming) or not. A "null" value signifies a non-trivial repair, while an array indicates a trivial repair and includes the associated trivial repair types.<b>call_graphs.json</b>The <i>call_graphs.json</i> file contains call graphs, generated through static code analysis, for test cases identified in the corresponding test repair commits. The file has a nested structure: at the first layer, each key represent a commit hash; at the second layer, each key is a qualified test case name; and at the third layer, it includes the call graph for the respective commit and the test case. Attributes of each call graph are as follows:<b>root </b>(Object): Represents the root node of the call graph, always the test case.<b>nodes </b>(Array): An array containing metadata for all nodes in the call graph.<b>graph </b>(Object): Represents edges of the graph, where each key corresponds to a node ID, and the associated value is an array of node IDs representing the nodes to which the key has outgoing edges.<b>sut_method_changes.json and sut_class_changes.json</b>Both files share a common structure, detailing code changes in the SUT for test repair commits. The <i>sut_method_changes.json</i> file exclusively captures changes within methods, while <i>sut_class_changes.json</i> covers all changes within classes. Both files present an array of objects, each object denoting changes in a commit. These objects have the following attributes:<b>bCommit, aCommit </b>(String): Same as <i>dataset.json</i>.<b>changes</b> (Array): An array of objects, each representing a change in a method or a class. These objects have the following attributes:<b>bPath, aPath</b> (String): Same as <i>dataset.json</i>.<b>name</b> (String): The qualified name of the changed method or the class.<b>hunks </b>(Array): An array of all hunks, representing changes in the method or class. The hunk data structure here follows that of the <i>hunk</i> field in <i>dataset.json.</i><b>is_test_source </b>(Boolean): Indicates whether the method or class is part of the project's test source code.<b>TaRGet Results</b>In addition to TaRBench, we provide the results of our approach, TaRGet. Below are the details of the results and associated files:<b>TaRGet_Results.zip</b>This ZIP archive contains two main folders:<b>Best_on_TaRBench</b><br>This folder includes the best results obtained using the best configuration of TaRGet on TaRBench during our experiments, i.e., IO2 with CodeT5+ model. Detailed information about the experimental setup and the optimal configuration can be found in our paper. The folder contains the following files:<b>train.json</b>, <b>valid.json</b>, <b>test.json</b>: These files represent the dataset splits for TaRBench. Each file includes the input and expected output of the model, formatted using the best configuration, along with the original TaRBench fields (described in detail above).<b>test_predictions.json</b>: This file contains the predictions generated by the optimal TaRGet configuration using beam search. Each entry includes:<b>ID</b>: The TaRBench ID.<b>target</b>: The expected output.<b>preds</b>: TaRGet's generations.<b>test_verdicts.json</b>: This file presents the results of executing the repairs generated by TaRGet (listed in test_predictions.json). Each entry includes:<b>Note</b>: A zero execution time indicates that the generated repair was identical to the correct repair and was skipped to save time.<b>ID</b>: The TaRBench ID.<b>rank</b>: The rank of the executed generation within the preds field of the predictions file.<b>verdict</b>: The execution result.<b>success</b>: A Boolean indicating whether the result was successful.<b>exec_time</b>: The execution time in seconds.<b>VS_CEPROT</b><br>This folder contains the results of the comparative analysis between TaRGet and CEPROT, a baseline test repair method used in our study. For further details about the comparison, refer to our paper. The folder includes:<b>test.json</b>: The test dataset in the same format as the dataset splits mentioned above. This dataset was used as the evaluation benchmark in CEPROT's study. Note that the <b>CID</b> field in this file, as well as in the subsequent files, refers to CEPROT's ID. This ID consists of two parts: the <b>focal_db ID</b> and the <b>test_db ID</b>, separated by a dash. The <b>ID</b> field, on the other hand, corresponds to a TaRBench-like ID that was created during the collection of CEPROT's data.<b>TaRGet_predictions.json</b>: The repairs generated by TaRGet on the test dataset.<b>CEPROT_predictions.json</b>: The repairs generated by CEPROT on the same test dataset.<b>TaRGet_Best_FineTuned_Model.zip</b>This ZIP archive contains the fine-tuned model of the optimal TaRGet configuration, which was used to generate the results in the "Best_on_TaRBench" folder. The archive includes:<b>checkpoint-best</b> folder: Contains the model files, including the fine-tuned model weights (pytorch_model.bin).<b>tokenizer </b>folder: Includes the tokenizer files used for the model, along with all additional special tokens.<b>Conclusion</b>TaRBench serves as a valuable resource for researchers, developers, and practitioners interested in automated test case repair. Through an evaluation on a diverse dataset, it provides insights into the capabilities and limitations of TaRGet, paving the way for advancements in the field of automated software testing and maintenance.