RAG-Coder: A Framework for Augmenting Qualitative Analysis in Empirical Software Engineering - Evaluation Dataset

📘 Description This repository contains the dataset used for the empirical evaluation of the RAG-Coder framework, as described in the paper: “RAG-Coder: A Framework for Augmenting Qualitative Analysis in Empirical Software Engineering” 2025. The dataset includes anonymized qualitative survey data and derived analysis files used to assess the reliability, performance, and interpretability of the RAG-Coder approach for coding open-ended responses. Due to company privacy and compliance requirements, this dataset cannot be publicly distributed.It can, however, be shared with authorized researchers upon reasonable request and with explicit authorization from the partner company. Requests should be directed to the corresponding author. 📁 Repository Structure / ├── raw_survey_data.csv # Original anonymized survey responses (restricted) ├── study1.csv # Manually Coded Dataset from Study 1 ├── study2.csv # Manually Coded Dataset from Study 2 ├── dataset_varinha_consenso.csv # Human consensus dataset used for model evaluation ├── adjudication_varinha.csv # Initial adjudication results between coders ├── adjudication_varinha_erros.csv # Annotated disagreements subset ├── rq1_metrics.json # Agreement metrics (RQ1) ├── rq2_results-ragcoder.csv # Framework results (RQ2) ├── rq3_adjudication.csv # Adjudicated disagreements (RQ3) ├── RAG-coding/ # RAG-Coder outputs for 775 responses │ ├── s1/ ... s4/ # Segment folders containing RAG-generated coding └── codebook.csv # Code definitions used for analysis 🔍 Contents Summary File/Folder Description raw_survey_data.csv Raw, anonymized survey responses from practitioners. study1.csv Dataset from the first study, now used by the RAG-Coder. study2.csv Dataset from the second study, now used by the RAG-Coder. dataset_varinha_consenso.csv Consensus codes from human analysts (gold standard). adjudication_varinha.csv Adjudication between first and second human coders. adjudication_varinha_erros.csv Subset of disagreement cases for detailed examination. rq1_metrics.json Cohen’s κ and other inter-coder agreement metrics. rq2_results-ragcoder.csv Output codes assigned by RAG-Coder for RQ2 validation. rq3_adjudication.csv Adjudicated disagreements between model and human coders (RQ3). RAG-coding/ RAG-Coder generated code assignments for all 775 responses. s1–s4/ Subsets representing RAG-Coder batch runs. codebook.csv Formal definitions of all qualitative codes. 🔒 Data Availability The dataset is restricted to protect participant privacy and company confidentiality.Access can be granted upon reasonable request and with company authorization. Interested researchers may contact the corresponding author to obtain access under a confidentiality agreement. ⚙️ Related Resources 🧠 RAG-Coder Framework (Open-Source): https://zenodo.org/records/17494982 📬 Contact Corresponding Author:Dr Lidiany CerqueiraEmail: [e-mail]: 0000-0002-4989-0986 📄 This dataset is provided exclusively for peer-review and research validation purposes under restricted access conditions.