Research Artifact – Cross-Project Flakiness: A Case Study of the OpenStack Ecosystem

This repository serves as the research artifact for the paper “Cross-Project Flakiness: A Case Study of the OpenStack Ecosystem”. It contains a dataset of flaky tests mined from OpenStack, along with their associated metadata, scripts for data collection, and the analysis logic to reproduce the findings in RQs 1–3. Requirements & Warnings Storage: This replication package is over 20 GB. Ensure you have sufficient disk space and a stable internet connection before downloading. Hardware: Due to the size of the datasets, high-memory (RAM) is recommended for running the analysis notebook. Database: A working installation of MongoDB and the mongorestore utility is required. Replication Workflow To replicate the results presented in the paper, follow these steps in order: Extract Logs: Unzip zuul_job_logs_2024_mark_filtered.zip into the project root. Restore Database: Use the mongorestore tool to import the study data from mongodb_archive.zip. Run Analysis: Execute scripts/BuildingResults.ipynb. This notebook contains all the logic to process the data and replicate the results, figures, and tables presented in the paper. Note: scripts/detect_flaky_tests.py is the script used for daily data collection. You do not need to run this for replication, as the output is already provided in the output/ folders. Repository Structure scripts/ – Data Collection and Analysis BuildingResults.ipynb: The main Jupyter notebook used to replicate key results. Note: This script is resource-intensive and may take a significant amount of time to complete. detect_flaky_tests.py: The script used to collect flaky tests on a daily basis. data/ – Dataset Files flaky_tests.json: All flaky tests identified in the study. cleaned_flaky_tests.json: Flaky tests that appeared in more than one patch set. flaky_builds_with_inline_comments.json: Flaky builds annotated with inline comments. flaky_jobs_with_inline_comments.json: Flaky jobs annotated with inline comments. inconsistent_flakiness_sample.xlsx: Manually labeled sample used for RQ3 analysis. output/ Output files generated by the detect_flaky_tests.py script. materials/ Visualizations and figures used in the paper. survey/ Questionnaire design and developer responses. External Data Archives (stored separately) zuul_job_logs_2024_mark_filtered.zip: A collection of job logs gathered during the study. mongodb_archive.zip: A MongoDB database dump containing data used in this study. Replication on the Zuul Ecosystem zuul_replication.zip: Scripts and data to replicate our results on the Zuul ecosystem.