Replication Package for "Chatting on flaky tests with standard LLMs. An empirical exploration"

Flaky tests yield inconsistent results without code changes and significantly impact software development costs, emphasizing the importance of effective detection methods. Despite various research efforts over the past 15 years, existing techniques often show limited adoption. This study explores the use of Large Language Models (LLMs), to detect flaky tests in software. Using the International Dataset of Flaky Tests, we asked commercially available LLMs, including GPT and Gemini, to classify Java test cases as flaky or non-flaky. Our results show that LLMs are unable to consistently and effectively identify flaky tests, indicating the need for alternative detection strategies. This research underscores the challenges of using LLMs for flaky test detection and highlights the need for more effective solutions. This archive contains the data we used for this experiment along with the artifacts obtained during our work on it. The ZIP file contains a description of the individual files in the package, which includes both the data and the Python scripts we used.