Research Artifacts of Validity Challenges in the Replication in Software Engineering

📋 Overview

This study aims to investigate how replication studies in Software Engineering report and address Threats to Validity (TTVs) from original studies, whether they reinterpret or reassess these threats, and whether new TTVs emerge during replication. This analysis highlights mitigation strategies and underscores the role of replication in enhancing methodological rigor, reliability, and external validity in empirical research.

To support this investigation and ensure transparency and reproducibility, this repository contains the dataset, scripts, and documentation associated with the paper Validity Challenges in the Replication in Software Engineering.

📂 Repository Structure

MitigationStrategies/
├── data/
│ ├── material/
│ │ ├── criteria.md
│ │ ├── materials.md
│ │ ├── scopus-results.xlsx
│ │ ├── scopus2022.bib
│ │ ├── scopus2023.bib
│ │ └── scopus2024.bib
│ ├── scripts/
│ │ ├── scripts.md
│ │ ├── MitigationStrategiesScript.R
│ │ ├── CorrMitTTVsMacroCategories.xlsx
│ │ ├── heatmap.jpg
│ │ └── heatmap.pdf
│ ├── SLR-data/
│ │ ├── data-details.md
│ │ └── data.xlsx
│ └── MitigationStrategies-Data.xlsx
├── LICENSE
├── LICENSE-MIT
├── Paper.pdf (This file will be added in the future)
└── README.md

📂 data/

Contains all materials, data, and scripts used or produced in the study.

> Download files

MitigationStrategies-Data.xlsx: This file contains the categorized qualitative data used to analyze mitigation strategies for TTVs in replication studies. The categories were derived through Reflexive Thematic Analysis and form the basis for the results presented in the paper.

📂 data/material/

Research planning material: search string, inclusion/exclusion criteria, and SCOPUS exports.

> Download files

scopus2022.bib: BibTeX file with all search results retrieved in 2022.

scopus2023.bib: BibTeX file with all search results retrieved in 2023.

scopus2024.bib: BibTeX file with all search results retrieved in 2024.

scopus-results.xlsx - Spreadsheet listing all retrieved studies (from 2022–2024) along with their inclusion or exclusion status from the selection process.

📂 data/scripts/

R scripts and supporting files for generating the figure and performing statistical analysis.

> Download files

MitigationStrategiesScript.R: R script to compute, through Fisher’s test and Phi Coefficient, the association between the different types of TTVs (internal, external, construct, and conclusion) and the four mitigation categories: Refinement and validation of data collection instruments (C1), Minimization of researcher bias and subjectivity (C2), Management and validation of data repositories/tools (C3), and Control and standardization of the experimental environment (C4). This analysis supports Figure 4 in the paper, which visualizes the associations between TTVs and mitigation categories.

CorrMitTTVsMacroCategories.xlsx: Data to run the R script (this sheet is also available in the spreadsheet MitigationStrategies-Data.xls).

heatmap.jpg and heatmap.pdf: Outputs of the R script.

📂 data/SLR-data/

Extracted data, quantitative, and qualitative summaries from the SLR process.

> Download files

data.xlsx: Spreadsheet containing all data points extracted during the SLR. It includes metadata about the selected studies, reported TTVs, and other coded variables used in the analysis.

📄 Paper.pdf

The preprint will be shared following approval.

📄 LICENSE / LICENSE-MIT

Define the usage terms for the data and scripts.

📄 README.md

This file provides an overview of the repository and a guide for usage.

💾 Storage Requirements

The total size of this repository is under 20 MB. No special storage requirements are needed.

🛠 How to Run the Scripts

⚙️ Requirements

To run the scripts, you need:

R installed on your machine.

RStudio (optional, but recommended).

🖥️ System Requirements

No specific hardware requirements are needed to execute the scripts.

🛠 Steps to Run

Download and install R.

Download and install RStudio (optional).

Download the R script from the scripts folder.

Open the R script.

Run the script using:

Mac: Cmd + Shift + Enter

Windows/Linux: Ctrl + Shift + Enter

The output is the statistical result and a plot, that will be saved in the same folder as the script.

💡 The script generates the Figure 4, presented in the paper.

💡 Make sure both the data file and R script are in the same directory.

🛡️ Ethical and Legal Considerations

This study was conducted strictly for academic research purposes and adheres to institutional research ethics guidelines. No personal or sensitive data were collected.

📜 License

R scripts are licensed under the MIT License.

Research data and artifacts are licensed under the CC BY 4.0 License.

Desafios de validade na replicação em engenharia de software