Research Artifacts of "Managing Threats to Validity in Software Engineering Replication Studies: An Evidence-Based Framework"

📋 Overview This artifact package provides all materials supporting the evidence-based framework proposed in this paper, including datasets, extraction forms, analysis scripts, coding materials, and traceability links between empirical findings and the resulting recommendations. The artifacts were produced across three complementary empirical investigations (Survey, Systematic Literature Review, and Qualitative Analysis) and are intended to facilitate transparency, reproducibility, independent verification, and future extensions of this research on Threats to Validity in Software Engineering replication studies. To support transparency and reproducibility, this repository contains the datasets, analysis scripts, focus group transcripts, and supporting materials associated with the paper Managing Threats to Validity in Software EngineeringReplication Studies: An Evidence-Based Framework. 📂 Repository Structure Focus Group Validation of Replication Recommendations/ Focus Group Validation of Replication Recommendations/├── data/ │ ├── material/ │ │ ├── Pilot transcript.pdf │ │ ├── Focus Group Transcript.pdf │ │ ├── Recommendations for Replication Studies.pdf │ │ ├── Focus Group Script │ ├── scripts/ │ │ ├── DotPlot.pdf │ │ ├── DotPlot.R │ │ ├── KrippendorffsAlpha.R │ ├── Focus Group-data/ │ │ └── Focus Group Validation of Replication Recommendations.xlsx├── LICENSE├── LICENSE-MIT├── Paper.pdf (This file will be added in the future)└── README.md 📂 data/ Contains all datasets, extraction forms, coding materials, traceability matrices, and analysis scripts used or produced throughout the empirical investigations that underpin the evidence-based framework for managing Threats to Validity in Software Engineering replication studies. This directory centralizes the empirical evidence supporting the analysis reported in the paper. 📂 data/material/ Contains the materials used to plan, conduct, and document the focus group study. > Download files Pilot transcript.pdfTranscript of the pilot focus group session conducted to refine the discussion protocol and validate the clarity and relevance of the guideline items. Focus Group Transcript.pdfAnonymized transcript of the main focus group session, which serves as the primary qualitative dataset for the analysis. Recommendations for Replication Studies.pdfThe version of the replication Recommendations that was evaluated and discussed by the participants during the focus group. Focus Group ScriptThe discussion guide used by the moderator, including prompts, questions, and facilitation structure adopted during the focus group sessions. 📂 data/scripts/ Contains scripts used to support quantitative summaries and reliability analysis derived from the focus group data. > Download files DotPlot.RR script used to generate dot plot visualizations summarizing participants’ assessments of the replication Recommendations. DotPlot.pdfOutput figure generated by the DotPlot.R script. KrippendorffsAlpha.RR script used to compute Krippendorff’s Alpha, supporting the assessment of inter-rater agreement in the quantitative evaluations produced during the focus group analysis. 📂 data/Focus Group-data/ Contains structured datasets derived from the focus group discussions, combining quantitative ratings and qualitative annotations. > Download files Focus Group Validation of Replication Recommendations.xlsxSpreadsheet consolidating participants’ evaluations of each Recommendation, including ratings, comments, and coding used in the analysis. This file represents the primary structured dataset used to synthesize the focus group results reported in the paper. 📄 Paper.pdf The studies that provide the empirical foundation for this research are listed below. 1 - Survey-Based Insights into the Replication Crisis and the 3R in Software Engineering: https://doi.org/10.5753/sbes.2025.9967 2 - Reimagining Studies’ Replication: A Validity-Driven Analysis of Threats in Empirical Software Engineering: https://doi.org/10.5753/sbes.2025.11270 3 - How Replication Studies Address Threats to Validity in Software Engineering: Under Review (The cited study is currently under review. For transparency, a preliminary version of the manuscript is included in the document JSERD_2026_Ivanildo.pdf) 📄 Paper.pdf The preprint will be shared following approval. 📄 LICENSE / LICENSE-MIT Define the usage terms for the data and scripts. 📄 README.md This file provides an overview of the repository and a guide for usage. 💾 Storage Requirements The total size of this repository is under 20 MB. No special storage requirements are needed. 🛠 How to Run the Scripts ⚙️ Requirements To run the scripts, you need: R installed on your machine. RStudio (optional, but recommended). 🖥️ System Requirements No specific hardware requirements are needed to execute the scripts. 🛠 Steps to Run Download and install R. Download and install RStudio (optional). Download the R script from the scripts folder. Open the R script. Run the script using: Mac: Cmd + Shift + Enter Windows/Linux: Ctrl + Shift + Enter The output is the statistical result and a plot, that will be saved in the same folder as the script. 💡 The script generates the Figure 4, presented in the paper. 💡 Make sure both the data file and R script are in the same directory. 🛡️ Ethical and Legal Considerations This study was conducted strictly for academic research purposes and adheres to institutional research ethics guidelines. No personal or sensitive data was made available. 📜 License R scripts are licensed under the MIT License. Research data and artifacts are licensed under the CC BY 4.0 License.