Do Automatic Test Generation Tools Generate Flaky Tests?

Published at the 46th International Conference on Software Engineering (ICSE 2024). Here you can find a preprint.About the artifacts<b>dataset.csv.gz</b>each row represents one test casecolumn "test_type": was the generated or developer-writtencolumn "flaky": has the test shown flaky behavior, and what kind? (NOD = non-order-dependent, OD = order-dependent)used to answer RQ1 (Prevalence) and RQ2 (Flakiness Suppression).<b>LoC.zip</b>contains lines-of-code data for the Java and Python projects<b>flaky_java_projects.zip</b> and <b>flaky_python_projects.zip</b>archives containing the 418 Java and 531 Python projects that contained at least one flaky testeach project contains the developer written and generated test suites<b>manual_rootCausing.zip</b>results of the manual root cause classificationfull_sample.csvcolumn "rater": which of the four researchers conducting the classification rated this test (alignment = all four)used to answer RQ3 (Root Causes)Running the jupyter notebookDownload all artifactsCreate and activate virtual environment<br><code>virtualenv -p venv</code><br><code>source venv/bin/activate</code>Install dependencies<br><code>pip install -r requirements.txt</code>Start jupyter lab<br><code>python -m jupyter lab</code>Scripts used for test generation and execution<br>Java (EvoSuite)Python (Pynguin)