Research Artifacts of Reimagining Studies' Replication: A Validity-Driven Analysis of Threats in Empirical Software Engineering

Research artifacts

The archived artifact is also publicly available via GitHub.

📋 Overview

This repository contains the dataset, scripts, and documentation for the paper Reimagining Studies' Replication: A Validity-Driven Analysis of Threats in Empirical Software Engineering published at Proceedings of the 39th Brazilian Symposium on Software Engineering - Insightful Ideas and Emerging Results (IIER '25) in 2025.

The study is a partial result of a Systematic Literature Review (SLR) that investigates how replication studies in Software Engineering address Threats to Validity (TTVs) of original and replication studies. It examines how these threats are identified and reported, as well as the implications for the reliability and generalizability of replicated results.

📂 Repository Structure

SLR2025/
├── data/
│ └── material/
│ └── scripts/
│ └── SLR-data/
├── LICENSE
├── LICENSE-MIT
├── Paper.pdf
├── README.md

📂 data/

Contains all materials, data, and scripts used or produced in the study.

📂 data/material/

Research planning material: search string, inclusion/exclusion criteria, SCOPUS exports, and threats to validity.

📂 data/scripts/

R scripts and supporting files for generating figures and performing statistical analysis.

📂 data/SLR-data/

Extracted data and quantitative summaries from the SLR process.

📄 Paper.pdf

The full paper, related to this artifact, describing the study's goals, methodology, analysis, and findings.

📄 LICENSE / LICENSE-MIT

Define the usage terms for the data and scripts.

📄 README.md

This file — includes an overview, usage guide, and citation instructions.

💾 Storage Requirements

The total size of this repository is under 20 MB. No special storage requirements are needed.

🛠 How to Run the Scripts

⚙️ Requirements

To run the scripts, you need:

R installed on your machine.

RStudio (optional, but recommended).

🖥️ System Requirements

No specific hardware requirements are needed to execute the scripts.

🛠 Steps to Run

Download and install R.

Download and install RStudio (optional).

Download an R script from the scripts folder.

Open an R script.

Open only one script at a time.

Run the script using:

Mac: Cmd + Shift + Enter

Windows/Linux: Ctrl + Shift + Enter

The output (e.g., plots) will be saved in the same folder as the script, using the same filename.

💡 Each script generates one figures used in the paper.

💡 For the Matthews Correlation Coefficient (MCC) script, ensure both the data file and R script are in the same directory.

📄 Citation

📝 Citing the Paper

If you use the data or findings from this repository, please cite the paper:

@inproceedings{AzevedoEtAl2025,
author = {Ivanildo Azevedo and
Ana Paula Vasconcelos and
Eudis Teixeira and
Sergio Soares},
title = {Reimagining Studies’ Replication: A Validity-Driven Analysis of Threats in Empirical Software Engineering},
booktitle = {Proceedings of the 39th Brazilian Symposium on Software Engineering - Insightful Ideas and Emerging Results (IIER '25)},
location = {Recife/PE},
year = {2025},
issn = {},
pages = {},
publisher = {SBC},
address = {Porto Alegre, RS, Brasil},
doi = {},
url = {}
}

📦 Citing the Artifact

If you prefer to cite only the archived artifact at Zenodo:

@misc{AzevedoEtAl2025,
author = {Ivanildo Azevedo and
Ana Paula Vasconcelos and
Eudis Teixeira and
Sergio Soares},
title = {Research Artifacts of Reimagining Studies’ Replication:
A Validity-Driven Analysis of Threats in Empirical Software Engineering},
year = {2025},
publisher = {Zenodo},
doi = {10.5281/zenodo.15511661},
url = {https://doi.org/10.5281/zenodo.15511661}
}

🛡️ Ethical and Legal Considerations

This study was conducted strictly for academic research purposes and adheres to institutional research ethics guidelines. No personal or sensitive data were collected.

📜 License

R scripts are licensed under the MIT License.

Research data and artifacts are licensed under the CC BY 4.0 License.