Replication Package for "Can Large Language Models Judge and Validate Qualitative Analysis? An Experience Report"

This repository contains the replication package for the paper "Can Large Language Models Judge and Validate Qualitative Analysis? An Experience Report".The artifact includes datasets, analysis scripts, statistical outputs, required to reproduce the results reported in the manuscript as well as all the already generated figures. The package is organized according to the research questions investigated in the study (RQ1--RQ4). Repository Structure . ├── DATA ├── FIGURES ├── RESULTS └── SCRIPTS DATA The DATA directory contains the input datasets used in the analyses. DATA/ ├── Input_1-3.csv └── Input_4.csv Input_1-3.csv Dataset used for the analyses related to: RQ1 RQ2 RQ3 Input_4.csv Dataset used for the analyses related to: RQ4 The files represent the raw input data used by the analysis scripts as collected according to the paper from the original study. SCRIPTS The SCRIPTS directory contains the Python scripts required to reproduce the analyses. Scripts are organized by research question. SCRIPTS/ ├── RQ1-2 ├── RQ3 └── RQ4 Each subdirectory contains: Analysis.pyMain script that executes the statistical analysis pipeline. Plots.pyScripts that generate the visualizations used in the paper. requirements.txtPython dependencies required to execute the scripts. run.shHelper script that runs the analysis pipeline. README.mdAdditional documentation specific to the RQ1--RQ2 analyses. RESULTS The RESULTS directory contains all outputs generated by the analysis scripts. RESULTS/ ├── RQ1-2 ├── RQ3 └── RQ4 Outputs include: summary statistics agreement metrics statistical test results model outputs intermediate analysis tables evaluation results These files correspond to the tables and statistical analyses reported in the paper. FIGURES The FIGURES directory contains the visualizations generated from the analyses. FIGURES/ ├── RQ1-2 ├── RQ3 └── RQ4 The directory includes: heatmaps boxplots distribution plots Likert visualizations forest plots metric distribution plots Figures are provided in PDF and PNG formats when applicable. Reproducing the Results Each research question can be reproduced independently. Run the Analysis Each directory includes a helper script run.sh that executes the analysis pipeline. Example: cd SCRIPTS/RQX bash run.sh The script performs the following steps: Install dependencies Loads the input data from the DATA directory Executes the statistical analyses Produces output tables in the RESULTS directory Generates figures in the FIGURES directory Requirements The artifact requires: Python 3.9 or newer Dependencies are listed in each requirements.txt file. Typical libraries include: pandas numpy scipy statsmodels matplotlib seaborn scikit-learn Reproducibility Notes All analyses are deterministic given the provided datasets.The repository contains all intermediate outputs required to reproduce the results reported in the paper. Scripts allow independent reproduction of each research question. Licenses Two licenses govern this artifact. Script License All scripts located in the SCRIPTS directory are distributed under theMIT License. See: SCRIPT_LICENSE Data License All datasets located in the DATA directory are distributed under theCreative Commons Attribution 4.0 International (CC BY 4.0) license. See: DATA_LICENSE