Research Artifacts of Guidelines for Addressing Threats to Validity in Replication Studies: Insights from a Focus Group

📋 Overview This study investigates how Software Engineering researchers collectively perceive and validate proposed replication guidelines. Through focus group analysis, it examines which guidelines are accepted, refined, or contested, and why, revealing key tensions between standardization and contextual flexibility in replication practices. To support this investigation and ensure transparency and reproducibility, this repository contains the dataset, scripts, and documentation associated with the paper Guidelines for Addressing Threats to Validity in ReplicationStudies: Insights from a Focus Group. 📂 Repository Structure MitigationStrategies/ ├── data/ │ ├── material/ │ │ ├── criteria.md │ │ ├── materials.md │ │ ├── scopus-results.xlsx │ │ ├── scopus2022.bib │ │ ├── scopus2023.bib │ │ └── scopus2024.bib │ ├── scripts/ │ │ ├── scripts.md │ │ ├── MitigationStrategiesScript.R │ │ ├── CorrMitTTVsMacroCategories.xlsx │ │ ├── heatmap.jpg │ │ └── heatmap.pdf │ ├── SLR-data/ │ │ ├── data-details.md │ │ └── data.xlsx │ └── MitigationStrategies-Data.xlsx ├── LICENSE ├── LICENSE-MIT ├── Paper.pdf (This file will be added in the future) └── README.md 📂 data/ Contains all materials, data, and scripts used or produced in the study. > Download files MitigationStrategies-Data.xlsx: This file contains the categorized qualitative data used to analyze mitigation strategies for TTVs in replication studies. The categories were derived through Reflexive Thematic Analysis and form the basis for the results presented in the paper. 📂 data/material/ Research planning material: search string, inclusion/exclusion criteria, and SCOPUS exports. > Download files scopus2022.bib: BibTeX file with all search results retrieved in 2022. scopus2023.bib: BibTeX file with all search results retrieved in 2023. scopus2024.bib: BibTeX file with all search results retrieved in 2024. scopus-results.xlsx - Spreadsheet listing all retrieved studies (from 2022–2024) along with their inclusion or exclusion status from the selection process. 📂 data/scripts/ R scripts and supporting files for generating the figure and performing statistical analysis. > Download files MitigationStrategiesScript.R: R script to compute, through Fisher’s test and Phi Coefficient, the association between the different types of TTVs (internal, external, construct, and conclusion) and the four mitigation categories: Refinement and validation of data collection instruments (C1), Minimization of researcher bias and subjectivity (C2), Management and validation of data repositories/tools (C3), and Control and standardization of the experimental environment (C4). This analysis supports Figure 4 in the paper, which visualizes the associations between TTVs and mitigation categories. CorrMitTTVsMacroCategories.xlsx: Data to run the R script (this sheet is also available in the spreadsheet MitigationStrategies-Data.xls). heatmap.jpg and heatmap.pdf: Outputs of the R script. 📂 data/SLR-data/ Extracted data, quantitative, and qualitative summaries from the SLR process. > Download files data.xlsx: Spreadsheet containing all data points extracted during the SLR. It includes metadata about the selected studies, reported TTVs, and other coded variables used in the analysis. 📄 Paper.pdf The preprint will be shared following approval. 📄 LICENSE / LICENSE-MIT Define the usage terms for the data and scripts. 📄 README.md This file provides an overview of the repository and a guide for usage. 💾 Storage Requirements The total size of this repository is under 20 MB. No special storage requirements are needed. 🛠 How to Run the Scripts ⚙️ Requirements To run the scripts, you need: R installed on your machine. RStudio (optional, but recommended). 🖥️ System Requirements No specific hardware requirements are needed to execute the scripts. 🛠 Steps to Run Download and install R. Download and install RStudio (optional). Download the R script from the scripts folder. Open the R script. Run the script using: Mac: Cmd + Shift + Enter Windows/Linux: Ctrl + Shift + Enter The output is the statistical result and a plot, that will be saved in the same folder as the script. 💡 The script generates the Figure 4, presented in the paper. 💡 Make sure both the data file and R script are in the same directory. 🛡️ Ethical and Legal Considerations This study was conducted strictly for academic research purposes and adheres to institutional research ethics guidelines. No personal or sensitive data were collected. 📜 License R scripts are licensed under the MIT License. Research data and artifacts are licensed under the CC BY 4.0 License. Desafios de validade na replicação em engenharia de software