LTM: Scalable and Black-box Similarity-based Test Suite Minimization based on Language Models - Replication Package

LTM: Scalable and Black-box Similarity-based Test Suite Minimization based on Language Models

This is the replication package associated with the paper "LTM: Scalable and Black-box Similarity-based Test Suite Minimization based on Language Models".

Replication Package Contents:

This replication package contains all the necessary data and code required to reproduce the results reported in the paper. We provide the results of the Fault Detection Rate (FDR), Total Minimization Time (MT), Time Saving Rate (TSR) , statistical tests for all the minimization budgets (i.e., 25%, 50%, and 75%), results for the preliminary study, results for UniXcoder/Cosine with preprocessed code on 16 projects.

Data:

We provide in the Data directory the data used in our experiments, which is the source code of test cases (Java test methods) of 17 projects collected from Defects4J.

Code:

We provide in the Code directory the code (Python) and bash files required to run the experiments and reproduce the results.

Results:

We provide in the Results directory the detailed results for our approach (called LTM). We also provide the summarized results of LTM and a baseline (ATM) for comparison purposes. Additional technical details about ATM can be found at https://zenodo.org/record/7455766.

_________________________________

LTM's Similarity Measurement:

The source code of this step is in the Code/LTM/Similarity directory.

Requirements:

To run this step, Python 3 is required (we used Python 3.10). Also, the required libraries in the Code/LTM/Similarity/requirements.txt file should be installed, as follows:

cd Code/LTM/Similarity

pip install -r requirements.txt

Input:

Data/LTM/TestMethods

Output:

Data/LTM/similarity_measurements

Running the experiment:

To measure the similarity between all pairs of test cases, the following bash script should be executed:

bash measure_similarity.sh

The source code of test methods of each project in the Data/LTM/TestMethods is parsed to generate pairs of test cases. This steps includes test methods tokenization, test methods embeddings extraction and similarity calculation. Then, all similarity scores are stored in Data/LTM/similarity_measurements folder. Due to the large size of the calculated similarity scores (60 GB), they were not uploaded on Zenodo, but they can be available upon request.

LTM's Test Suite Minimization:

The source code of this step is in the Code/LTM/Search directory.

Requirements:

To run this step, Python 3 is required (we used Python 3.10). Also, the required libraries in the Code/LTM/Search/requirements.txt file should be installed, as follows:

cd Code/LTM/Search

pip install -r requirements.txt

Input:

Data/LTM/similarity_measurements

Output:

Results/LTM/minimization_results

Running the experiments:

To minimize the test suite for each project version, the following bash script should be executed:

bash minimize.sh

The similarity scores of all test case pairs per project version are parsed by the search algorithm (Genetic Algorithm). Each experiment runs ten times using three minimization budgets (25%, 50%, and 75%). The results are stored in the Results/LTM/minimization_results directory.

LTM's Evaluation:

To evaluate the minimization results for each version and each project, the following bash script should be executed:

cd Code/LTM/Evaluation

bash evaluate_per_version.sh

cd Code/LTM/Evaluation

bash evaluate_per_project.sh

This will evaluate the FDR, MT and TSR results for each version and each project for each minimization budget. These results are stored in the Results/LTM directory.

Note that for each version, the FDR is either 1 or 0. For each project, the FDR ranges from 0 to 1.