RAG-Coder: A Framework for Augmenting Qualitative Analysis in Empirical Software Engineering - Evaluation Dataset

📘 Description This repository contains the dataset used for the empirical evaluation of the RAG-Coder framework, as described in the paper: “RAG-Coder: A Framework for Augmenting Qualitative Analysis in Empirical Software Engineering” 2025. The dataset includes anonymized qualitative survey data and derived analysis files used to assess the reliability, performance, and interpretability of the RAG-Coder approach for coding open-ended responses. Due to company privacy and compliance requirements, this dataset cannot be publicly distributed.It can, however, be shared with authorized researchers upon reasonable request and with explicit authorization from the partner company. Requests should be directed to the corresponding author. 📁 Repository Structure / ├── raw_survey_data.csv # Original anonymized survey responses (restricted) ├── study1.csv # Manually Coded Dataset from Study 1 ├── study2.csv # Manually Coded Dataset from Study 2 ├── dataset_varinha_consenso.csv # Human consensus dataset used for model evaluation ├── adjudication_varinha.csv # Initial adjudication results between coders ├── adjudication_varinha_erros.csv # Annotated disagreements subset ├── rq1_metrics.json # Agreement metrics (RQ1) ├── rq2_results-ragcoder.csv # Framework results (RQ2) ├── rq3_adjudication.csv # Adjudicated disagreements (RQ3) ├── RAG-coding/ # RAG-Coder outputs for 775 responses │ ├── s1/ ... s4/ # Segment folders containing RAG-generated coding └── codebook.csv # Code definitions used for analysis 🔍 Contents Summary File/Folder Description raw_survey_data.csv Raw, anonymized survey responses from practitioners. study1.csv Dataset from the first study, now used by the RAG-Coder. study2.csv Dataset from the second study, now used by the RAG-Coder. dataset_varinha_consenso.csv Consensus codes from human analysts (gold standard). adjudication_varinha.csv Adjudication between first and second human coders. adjudication_varinha_erros.csv Subset of disagreement cases for detailed examination. rq1_metrics.json Cohen’s κ and other inter-coder agreement metrics. rq2_results-ragcoder.csv Output codes assigned by RAG-Coder for RQ2 validation. rq3_adjudication.csv Adjudicated disagreements between model and human coders (RQ3). RAG-coding/ RAG-Coder generated code assignments for all 775 responses. s1–s4/ Subsets representing RAG-Coder batch runs. codebook.csv Formal definitions of all qualitative codes. 🔒 Data Availability The dataset is restricted to protect participant privacy and company confidentiality.Access can be granted upon reasonable request and with company authorization. Interested researchers may contact the corresponding author to obtain access under a confidentiality agreement. ⚙️ Related Resources 🧠 RAG-Coder Framework (Open-Source): https://zenodo.org/records/17494982 📬 Contact Corresponding Author:Dr Lidiany CerqueiraEmail: [e-mail]: 0000-0002-4989-0986 📄 This dataset is provided exclusively for peer-review and research validation purposes under restricted access conditions.📘 数据集说明 本仓库包含用于实证评估RAG-Coder框架的数据集，对应发表于2025年的论文《RAG-Coder：实证软件工程中的质性分析增强框架》。 本数据集包含匿名化的质性调研数据及衍生分析文件，用于评估RAG-Coder方法对开放式调研回应进行编码时的可靠性、性能与可解释性。 鉴于企业隐私与合规要求，本数据集不可公开发布。但经合理申请并获得合作企业明确授权后，可向获批研究人员共享。申请请求需提交至通讯作者。 📁 仓库结构 / ├── raw_survey_data.csv # 原始匿名化调研回复（受限访问） ├── study1.csv # 研究1的人工编码数据集 ├── study2.csv # 研究2的人工编码数据集 ├── dataset_varinha_consenso.csv # 用于模型评估的人工共识数据集 ├── adjudication_varinha.csv # 编码员间初步裁决结果 ├── adjudication_varinha_erros.csv # 标注的分歧案例子集 ├── rq1_metrics.json # 一致性指标（研究问题1） ├── rq2_results-ragcoder.csv # 框架实验结果（研究问题2） ├── rq3_adjudication.csv # 已裁决的分歧案例（研究问题3） ├── RAG-coding/ # 775条调研回应的RAG-Coder输出结果 │ ├── s1/ ... s4/ # 包含RAG生成编码结果的分段文件夹 └── codebook.csv # 分析所用的编码定义 🔍 内容概览 文件/文件夹 说明 raw_survey_data.csv 来自行业从业者的原始匿名化调研回复 study1.csv 第一项研究的数据集，现已用于RAG-Coder相关实验 study2.csv 第二项研究的数据集，现已用于RAG-Coder相关实验 dataset_varinha_consenso.csv 人工分析师生成的共识编码（金标准数据集） adjudication_varinha.csv 两名人工编码员间的初步裁决结果 adjudication_varinha_erros.csv 用于详细审查的分歧案例子集 rq1_metrics.json Cohen’s κ及其他编码员间一致性指标 rq2_results-ragcoder.csv RAG-Coder为验证研究问题2所分配的输出编码 rq3_adjudication.csv 模型与人工编码员间已裁决的分歧案例（研究问题3） RAG-coding/ 针对全部775条调研回应的RAG-Coder生成编码分配结果 s1–s4/ 代表RAG-Coder分批运行的子集 codebook.csv 所有质性编码的正式定义 🔒 数据可用性 本数据集受访问限制，以保护参与者隐私与企业机密。经合理申请并获得企业授权后，可获取数据集。有意愿的研究人员可联系通讯作者，在签署保密协议后获取访问权限。 ⚙️ 相关资源 🧠 RAG-Coder框架（开源）：https://zenodo.org/records/17494982 📬 联系方式 通讯作者：莉迪亚尼·塞尔凯拉（Lidiany Cerqueira）博士 邮箱：[e-mail] ORCID：0000-0002-4989-0986 📄 本数据集仅可在受限访问条件下，用于同行评审与研究验证工作。