Experimental data LLMProphet - AIST 2025 (co-located with ICST 2025)

Reproducing Results for LLMProphet - AIST 2025

Paper Information

Title: From Implemented to Expected Behaviors: Leveraging Regression Oracles for Non-Regression Fault Detection Using LLMs

Authors: Stefano Ruberto, Judith Perera, Gunel Jahangirova, Valerio Terragni

Affiliations:

Stefano Ruberto, JRC European Commission, Ispra, Italy ([e-mail])

Judith Perera, University of Auckland, Auckland, New Zealand ([e-mail])

Gunel Jahangirova, King’s College London, London, United Kingdom ([e-mail])

Valerio Terragni, University of Auckland, Auckland, New Zealand ([e-mail])

Published in: at the 5th International Workshop on Artificial Intelligence in Software Testing (AIST 2025) co-located with 18th IEEE International Conference on Software Testing, Verification and Validation (ICST) 2025

Reproduction Instructions

Requirements

To reproduce the results, you will need access to OpenAI's GPT-4o model.

Steps

Access GPT-4o:

Open the OpenAI Playground: GPT-4o Playground

Set the following parameters:

Temperature: 0.0

Top-p: 1.0

Run the prompts:

Copy and paste the prompts from the respective folders into the "User" prompt field, one at a time.

Make sure to clear the input before inserting a new prompt.

Prompt Folders:

Prompts_with_test_case_combinations_evaluating_19/05/22

Prompts_with_test_case_combinations_evaluating_15/05/22

File Naming Convention

Inside the folders, files are named as follows:

15_10_wrong_0_correct.java

This indicates: - 15/05/2022: The test case being evaluated. - 10_wrong: Number of test cases with faulty assertions. - 0_correct: Number of test cases with correct assertions. - .java: The files are stored in .java format to maintain formatting.

Notes

Ensure you follow the reproduction steps precisely to achieve the same results as presented in the paper.

The files inside the folders contain the necessary test cases for evaluation.

Contact

For any questions regarding this material, please reach out to the authors via the provided email addresses.