Supplemental material for: A Research Roadmap on the Usage of Large Language Models in Software Testing

This repository contains the replication package of the paper A Research Roadmap on the Usage of Large Language Models in Software Testing published at TO-DO

The replication package comprises the raw and the validation data used in the roadmap, as well as an interactive view hosted in GitHub Pages. In ZENODO, we made available the validation and final datasets curated by the different researchers in a CSV format, with a structure (columns) as follows:

The roadmap dataset (articlecorpus.csv) is a semicolon-separated CSV file, with a structure (columns) as follows:

ID: Internal reference used in the study, "PXX" for the original corpus and "VXX" for validation.

Title: Title of the article.

Year: Publication year or when it was made available on arXiv.

Key: BibTeX key.

Published into: Name of the journal or conference where it is published (prefixed with "C:" for conferences or "J:" for journals).

Publication type: Type of publication: Conference, Journal, or arXiv.

Bibtex: BibTeX entry of the publication.

Database: Source database: ACM, IEEE, Elsevier, Springer, or Orig.

Type of Contribution: Type of contribution: Survey, Evaluation, or Research Contribution.

Abstract: Short abstract retrieved from the article.

Trend: Type of LLM-based testing: Unit Test Generation, High-Level Test Generation, Oracle Derivation, Test Augmentation or Improvement, Test Configuration or Execution, and Reflections.

LLM Interaction: Type of LLM interaction: Pure Prompting or Hybrid Prompting.

Domain Specific Knowledge: How the LLM is improved with context: None, RAG, or Fine-Tuning.

Approach: Type of approach: Tool/Approach or Agent.

Scope: Testing scope: Functional or Non-Functional.

Focus: indicates whether the approach is intended to generate Code/Procedure, Data, or Optimization.

Benchmark: Name or identifier of the benchmark used.

LLMs Used: Name of the model/models used in the article.

Evaluation Metric: Name of the metrics used in evaluating the article.

Tool: Name of the tool proposed by the article.

The validation dataset (validation.csv) is a semicolon-separated CSV file, with a structure (columns) as follows:

ID: Internal reference used in the study.

Title: Title of the article.

Authors: Authors of the article.

Year: Publication year.

DOI: article DOI retrieved programmatically.

Abstract: Short abstract retrieved from the article.

I/E: result of the inclusion/exclusion criteria.

To cite this work: 

Cristian Augusto, Antonia Bertolino, Guglielmo De Angelis, Francesca Lonetti, and Jesús Morán, “A Research Roadmap on the Usage of Large Language Models in Software Testing”, Journal Name, XXX, YYY. https://doi.org/XXXXXX