ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolutionary Search – Replication Package

This is the replication package associated with the paper "ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolutionary Search" accepted at the 45th IEEE/ACM International Conference on Software Engineering (ICSE 2023) – Technical Track. Cite this paper using the following:

@inproceedings{pan2023atm,
  title={ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolutionary Search},
  author={Pan, Rongqi and Ghaleb, Taher A. and Briand, Lionel},
  booktitle={Proceedings of the 45th IEEE/ACM International Conference on Software Engineering},
  year={2023},
  pages={1--12}
}

Replication Package Contents:
The replication package contains all the necessary data and code required to reproduce the results reported in the paper. We also provide the results for other minimization budgets, and detailed FDR, execution time, and statistical test results. In addition, we provide the data and code required to reproduce the results of baselines techniques: FAST-R and random minimization.

Data:
We provide in the Data directory the data used in our experiments, which is based on 16 projects from Defects4J, whose characteristics can be found in Data/subject_projects.csv.

Code:
We provide in the Code directory the code and scripts (Java, Python, and Bash) required to run the experiments and reproduce the results.

Results:
We provide in the Results directory the results for each technique independently, and also a summary of all results together for comparison purposes. The source code for this step is in the Code/ATM/CodeToAST directory. The source code for this step is in the Code/ATM/Similarity directory.

_________________________________

ATM - Code to AST transformation:

Requirements:
* Eclipse IDE (we used 2021-12)
* The libraries (the .jar files in the Code/ATM/CodeToAST/lib directory)

Input:
All zipped data files should be unzipped before running each step.
* Data/test_suites/all_test_cases.zip → Data/test_suites/all_test_cases
* Data/test_suites/changed_test_cases.zip → Data/test_suites/changed_test_cases
* Data/test_suites/relevant_test_cases.zip → Data/test_suites/relevant_test_cases

Output:
* Data/ATM/ASTs/all_test_cases
* Data/ATM/ASTs/changed_test_cases

Running the experiment:
To generate ASTS for all test cases in the project test suites, the Code/ATM/CodeToAST/src/CodeToAST.java file should be compiled and run using the Eclipse IDE by including all the required .jar files in the Code/ATM/CodeToAST/lib directory as part of the classpath. A bash script is provided along with a pre-generated .jar file in the Code/ATM/CodeToAST/bin directory to run this step, as follows:

cd Code/ATM/CodeToAST
bash transform_code_to_ast.sh

Each test file in the Data/test_suites/all_test_cases and Data/test_suites/changed_test_cases directories is parsed to generate a corresponding AST for each test case method (saved in an XML format in Data/ATM/ASTs/all_test_cases and Data/ATM/ASTs/changed_test_cases for each project version)
_________________________________

ATM - Similarity Measurement:

Requirements:
* Eclipse IDE (we used 2021-12)
* The libraries (the .jar files in the Code/ATM/Similarity/lib directory)

Input:
* Data/test_suites/all_test_cases
* Data/test_suites/changed_test_cases

Output:
* Data/ATM/similarity_measurements

Running the experiment:
To measure the similarity between each pair of test cases, the Code/ATM/Similarity/src/SimilarityMeasurement.java file should be compiled and run using the Eclipse IDE by including all the required .jar files in the Code/ATM/Similarity/lib directory as part of the classpath. A bash script is provided along with a pre-generated .jar file in the Code/ATM/Similarity/bin directory to run this step, as follows:

cd Code/ATM/Similarity
bash measure_similarity.sh

ASTs of each project in the Data/ATM/ASTs/all_test_cases and Data/ATM/ASTs/changed_test_cases directories are parsed to create pairs of ASTs containing one test case from the Data/ATM/ASTs/all_test_cases directory with another test case from the Data/ATM/ASTs/changed_test_cases directory (redundant pairs are discarded). Then, all similarity measurements are saved in the Data/ATM/similarity_measurements.zip file.
__________________________________________

Search-based Minimization Algorithms:
The source code for this step is in the Code/ATM/Search directory.

Requirements:
To run this step, Python 3 is required (we used Python 3.10). Also, the libraries in the Code/AMT/Search/requirements.txt file should be installed, as follows:

cd Code/ATM/Search
pip install -r requirements.txt

Input:
* Data/ATM/similarity_measurements

Output:
* Results/ATM/minimization_results

Running the experiment:
To minimize the test suites in our dataset, the following bash script should be executed:

bash minimize.sh

All similarity measurements are parsed for each version of the projects, independently. Each version is run 10 times using three minimization budgets (25%, 50%, and 75%). Genetic Algorithm (GA) is run using four similarity measures, namely top-down, bottom-up, combined, and tree edit distance. NSGA-II is run using two combinations of similarity measures: top-down & bottom-up and combined & tree edit distance. The minimization results are generated in the Results/ATM/minimization_results directory.
__________________

Evaluate results:
To evaluate and summarize the minimization results, run the following:

cd Code/ATM/Evaluation
bash evaluate.sh

This will generate summarized FDR and execution time results (per-project and per-version) for each minimization budget, which can all be found in Results/ATM. In this replication package, we provide the final, merged FDR with execution time results.

_________________________________

Running FAST-R experiments
ATM was compared to FAST-R, a state-of-the-art baseline, which is a set of test case minimization techniques called: FAST++, FAST-CS, FAST-pw, and FAST-all, which we adapted to our data and experimental setup.

Requirements:
To run this step, Python 3.7 is required. Also, the libraries in the Code/FAST-R/requirements.txt file should be installed, as follows:

cd Code/FAST-R
pip install -r requirements.txt

Input:
* Data/FAST-R/test_methods
* Data/FAST-R/test_classes

Output:
* Results/FAST-R/test_methods/FDR_and_Exec_Time_Results_[budget]%_budget.csv
* Results/FAST-R/test_classes/FDR_and_Exec_Time_Results_[budget]%_budget.csv

To run FAST-R experiments, the following bash script should be executed:

bash fast_r.sh test_methods #method level
bash fast_r.sh test_classes #class level

Results are generated in .csv files for each budget. For example, for the 50% budget, results are saved in FDR_and_Exec_Time_Results_50%_budget.csv in the Results/FAST-R/test_methods and Results/FAST-R/test_classes directories.

_________________________________

Running the random minimization experiments
ATM was also compared to random minimization as a standard baseline.

Requirements: To run this step, Python 3 is required (we used Python 3.10). Also, the libraries in the Code/RandomMinimization/requirements.txt file should be installed, as follows:

cd Code/RandomMinimization
pip install -r requirements.txt

Input:
N/A

Output:
* Results/RandomMinimization/FDR_and_Exec_Time_Results_[budget]%_budget.csv

To run the random selection experiments, the following bash script should be executed:

bash random_minimization.sh

Results are generated in .csv files for each budget. For example, for the 50% budget, results are saved in FDR_and_Exec_Time_Results_50%_budget.csv in the Results/RandomMinimization directory.