Datasets and scripts related to the paper: "*Can Generative AI Help us in Open Coding of Software Engineering Data?*"

This replication package contains datasets and scripts related to the paper: "Can Generative AI Help us in Open Coding of Software Engineering Data?" The replication package is organized into two directories: manual_analysis: This directory contains all sheets used to perform the manual analysis for RQ1, RQ2, and RQ3. stats: This directory contains all datasets, scripts, and results metrics used for the quantitative analyses of RQ1 and RQ2. In the following, we describe the content of each directory: manual_analysis manual_analysis_rq1: This directory contains all sheets used to perform manual analysis for RQ1 (independent and incremental coding). The sub-directory incremental_coding contains .csv files for all datasets (DL_Faults_COMMIT_incremental.csv, DL_Faults_ISSUE_incremental.csv, DL_Fault_SO_incremental.csv, DRL_Challenges_incremental.csv and Functional_incremental.csv). All these .csv files contain the following columns: Link: The link to the instances Prompt: Prompt used as input to GPT-4-Turbo ID: Instance ID FinalTag: Tag assigned by the human in the original paper Chatgpt_output_memory: Output of GPT-4-Turbo with incremental coding Chatgpt_output_memory_clean: (only for the DL Faults datasets) output of GPT-4-Turbo considering only the label assigned, excluding the text Author1: Label assigned by the first author Author2: Label assigned by the second author FinalOutput: Label assigned after the resolution of the conflicts The sub-directory independent_coding contains .csv files for all datasets (DL_Faults_COMMIT_independent.csv, DL_Faults_ISSUE_ independent.csv, DL_Fault_SO_ independent.csv, DRL_Challenges_ independent.csv and Functional_ independent.csv), containing the following columns: Link: The link to the instances Prompt: Prompt used as input to GPT-4-Turbo ID: Specific ID for the instance FinalTag: Tag assigned by the human in the original paper Chatgpt_output: Output of GPT-4-Turbo with independent coding Chatgpt_output_clean: (only for DL Faults datasets) output of GPT-4-Turbo considering only the label assigned, excluding the text Author1: Label assigned by the first author Author2: Label assigned by the second author FinalOutput: Label assigned after the resolution of the conflicts. Also, the sub-directory contains sheets with inconsistencies after resolving conflicts. The directory inconsistency_incremental_coding contains .csv files with the following columns: Dataset: The dataset considered Human: The label assigned by the human in the original paper Machine: The label assigned by GPT-4-Turbo Classification: The final label assigned by the authors after resolving the conflicts. Multiple classifications for a single instance are separated by a comma “,” Final: final label assigned after the resolution of the incompatibilities Similarly, the sub-directory inconsistency_independent_coding contains a .csv file with the same columns as before, but this is for the case of independent coding. manual_analysis_rq2: This directory contains .csv files for all datasets (DL_Faults_redundant_tag.csv, DRL_Challenges_redundant_tag.csv, Functional_redundant_tag.csv) to perform manual analysis for RQ2. The DL_Faults_redundant_tag.csv file contains the following columns: Tags Redundant: tags identified as redundant by GPT-4-Turbo Matched: inspection by the authors to see if the tags are redundant matching or not FinalTag: final tag assigned by the authors after the resolution of the conflict The Functional_redundant_tag.csv file contains the same columns as before The DRL_Challenges_redundant_tag.csv file is organized as follows: Tags Suggested: The final tag suggested by GPT-4-Turbo Tags Redundant: tags identified as redundant by GPT-4-Turbo Matched: inspection by the authors to see if the tags redundant matching or not with the tags suggested FinalTag: final tag assigned by the authors after the resolution of the conflict The sub-directory code_consolidation_mapping_overview contains .csv files (DL_Faults_rq2_overview.csv, DRL_Challenges_rq2_overview.csv, Functional_rq2_overview.csv) organized as follows: Initial_Tags: list of the unique initial tags assigned by GPT-4-Turbo for each dataset Mapped_tags: list of tags mapped by GPT-4-Turbo Unmatched_tags: list of unmatched tags by GPT-4-Turbo Aggregating_tags: list of consolidated tags Final_tags: list of final tags after the consolidation task prompt_for_each_rq: This directory contains: - (i) the history of prompts used in each dataset (prompts_history.txt) -(ii) all final prompt used for the analysis of each dataset, prompt used for incremental coding, prompt used in rq2 to consolidate redundant codes, prompt used in rq3 to create taxonomy (generic_prompt.txt) -(iii) all .csv files in which there are indicate, for each dataset, the link and the prompt used (prompt_DL_Faults_COMMIT.csv, prompt_DL_Faults_ISSUE.csv, prompt_DL_Faults_SO.csv, prompt_DRL_Challenges.csv). For the Functional Dataset .csv file contains, instead, Question, Answer and Prompt used (prompt_Functional.csv) rq3: This directory contains the taxonomies obtained from GPT-4-Turbo for the DL Faults and for the DRL Challenges (taxonomy_DL_Faults.txt,taxonomy_DRL_Challenges.txt) stats RQ1: contains script and datasets used to perform metrics for RQ1. The analysis calculates all possible combinations between Matched, More Abstract, More Specific, and Unmatched. RQ1_Stats.ipynb is a Python Jupyter nooteook to compute the RQ1 metrics. To use it, as explained in the notebook, it is necessary to change the values of variables contained in the first code block. independent-prompting: Contains the datasets related to the independent prompting. Each line contains the following fields: Link: Link to the artifact being tagged Prompt: Prompt sent to GPT-4-Turbo FinalTag: Artifact coding from the replicated study chatgpt_output_text: GPT-4-Turbo output chatgpt_output: Codes parsed from the GPT-4-Turbo output Author1: Annotator 1 evaluation of the coding Author2: Annotator 2 evaluation of the coding FinalOutput: Consolidated evaluation incremental-prompting: Contains the datasets related to the incremental prompting (same format as independent prompting) results: contains files for the RQ1 quantitative results. The files are named RQ1\_<<Dataset>>\_<<Prompt method>>\_<<ExcludingNegative>>\_<<MetricAggregation>>.csv, where Dataset is the dataset name, Prompt method indicates whether results are for independent or incremental prompting, Excluding Negatives (for datasets where this applies) whether results have been obtained by excluding negative instances, and MetricAggregation (where it applies) how metrics have been aggregated (macro or weighted average). The files report columns indicating the Dataset, the Matching type, the Accuracy, Precision, Recall, F1 Score, and Cohen's Kappa. RQ2: contains the script used to perform metrics for RQ2, the datasets it uses, and its output. RQ2_SetStats.ipynb is the Python Jupyter notebook to perform the analyses. The scripts takes as input the following types of files, contained in the directory contains the script used to perform the metrics for RQ2. The script takes in input: RQ1 Data Files (RQ1_DLFaults_Issues.csv, RQ1_DLFaults_Commits.csv, and RQ1_DLFaults_SO.csv, joined in a single .csv RQ1_DLFaults.csv). These are the same files used in RQ1. Mapping Files (RQ2_Mappings_DRL.csv, RQ2_Mappings_Functional.csv, RQ2_Mappings_DLFaults.csv). These contain the mappings between human tags (HumanTags), GPT-4-Turbo tags (Final Tags), with indicated the type of matching (MatchType). Additional codes creating during the consolidation (RQ2_newCodes_DRL.csv, RQ2_newCodes_Functional.csv, RQ2_newCodes_DLFaults.csv), annotated with the matching: new code,old code,human code,match type Set files (RQ2_Sets_DRL.csv, RQ2_Sets_Functional.csv, RQ2_Sets_DLFaults.csv). Each file contains the following columns: HumanTags: List of tags from the original dataset InitialTags: Set of tags from RQ1, ConsolidatedTags: Tags that have been consolidated, FinalTags: Final set of tags (results of RQ2, used in RQ3) NewTags: New tags created during consolidation RQ2_Set_Metrics.csv: Reports the RQ2 output metrics (Precision, Recall, F1-Score, Jaccard).本复现包（replication package）包含与论文《生成式AI(Generative AI)能否助力软件工程数据的开放式编码(open coding)？》相关的数据集与脚本。 本复现包分为两个目录： manual_analysis：该目录包含用于开展RQ1、RQ2与RQ3手动分析的全部工作表。 manual_analysis_rq1：该目录包含用于开展RQ1手动分析（独立式编码(independent coding)与增量式编码）的全部工作表。 增量式编码子目录：包含各数据集的.csv文件，分别为DL_Faults_COMMIT_incremental.csv、DL_Faults_ISSUE_incremental.csv、DL_Fault_SO_incremental.csv、DRL_Challenges_incremental.csv与Functional_incremental.csv。所有.csv文件均包含以下列： - Link：实例的链接 - Prompt：输入至GPT-4-Turbo(GPT-4-Turbo)的提示词 - ID：实例ID - FinalTag：原论文中人工标注的标签 - Chatgpt_output_memory：采用增量式编码的GPT-4-Turbo输出结果 - Chatgpt_output_memory_clean：（仅适用于DL Faults数据集）仅保留标注结果、剔除文本内容的GPT-4-Turbo输出 - Author1：第一位标注者赋予的标签 - Author2：第二位标注者赋予的标签 - FinalOutput：冲突解决后确定的最终标签 独立式编码子目录：包含各数据集的.csv文件，分别为DL_Faults_COMMIT_independent.csv、DL_Faults_ISSUE_independent.csv、DL_Fault_SO_independent.csv、DRL_Challenges_independent.csv与Functional_independent.csv，包含以下列： - Link：实例的链接 - Prompt：输入至GPT-4-Turbo的提示词 - ID：实例的专属ID - FinalTag：原论文中人工标注的标签 - Chatgpt_output：采用独立式编码的GPT-4-Turbo输出结果 - Chatgpt_output_clean：（仅适用于DL Faults数据集）仅保留标注结果、剔除文本内容的GPT-4-Turbo输出 - Author1：第一位标注者赋予的标签 - Author2：第二位标注者赋予的标签 - FinalOutput：冲突解决后确定的最终标签 该子目录还包含冲突解决后存在不一致情况的工作表。其中inconsistency_incremental_coding子目录包含以下列的.csv文件： - Dataset：所用数据集 - Human：原论文中人工标注的标签 - Machine：GPT-4-Turbo赋予的标签 - Classification：作者解决冲突后确定的最终标签，单个实例的多分类结果以逗号", "分隔 - Final：解决不匹配问题后确定的最终标签 同理，inconsistency_independent_coding子目录包含与上述结构一致的.csv文件，对应独立式编码场景。 manual_analysis_rq2：该目录包含用于开展RQ2手动分析的各数据集的.csv文件，分别为DL_Faults_redundant_tag.csv、DRL_Challenges_redundant_tag.csv、Functional_redundant_tag.csv。 其中DL_Faults_redundant_tag.csv包含以下列： - Tags Redundant：GPT-4-Turbo识别出的冗余标签 - Matched：作者用于检查标签是否为冗余匹配项的核查项 - FinalTag：作者解决冲突后确定的最终标签 Functional_redundant_tag.csv包含与上述相同的列。 DRL_Challenges_redundant_tag.csv的结构如下： - Tags Suggested：GPT-4-Turbo提出的最终标签 - Tags Redundant：GPT-4-Turbo识别出的冗余标签 - Matched：作者用于核查冗余标签与建议标签是否匹配的核查项 - FinalTag：作者解决冲突后确定的最终标签 code_consolidation_mapping_overview子目录包含以下.csv文件：DL_Faults_rq2_overview.csv、DRL_Challenges_rq2_overview.csv、Functional_rq2_overview.csv，其结构如下： - Initial_Tags：各数据集由GPT-4-Turbo赋予的唯一初始标签列表 - Mapped_tags：GPT-4-Turbo映射后的标签列表 - Unmatched_tags：GPT-4-Turbo未匹配到的标签列表 - Aggregating_tags：整合后的标签列表 - Final_tags：标签整合任务完成后的最终标签列表 prompt_for_each_rq：该目录包含： 1. 各数据集所用提示词的历史记录（prompts_history.txt）； 2. 各分析任务所用的全部最终提示词，包括增量式编码所用提示词、RQ2中用于冗余代码整合的提示词、RQ3中用于构建分类体系的提示词（generic_prompt.txt）； 3. 各数据集对应的链接与所用提示词的全部.csv文件，分别为prompt_DL_Faults_COMMIT.csv、prompt_DL_Faults_ISSUE.csv、prompt_DL_Faults_SO.csv、prompt_DRL_Challenges.csv。针对功能数据集（Functional Dataset），其对应的.csv文件还包含问题、答案与所用提示词（prompt_Functional.csv）。 rq3：该目录包含由GPT-4-Turbo为DL Faults数据集与DRL Challenges数据集生成的分类体系（taxonomy_DL_Faults.txt、taxonomy_DRL_Challenges.txt）。 stats：该目录包含用于RQ1与RQ2定量分析的全部数据集、脚本与结果指标。 RQ1：包含用于计算RQ1指标的脚本与数据集。本分析针对匹配（Matched）、更抽象（More Abstract）、更具体（More Specific）与未匹配（Unmatched）的所有可能组合进行计算。 RQ1_Stats.ipynb为用于计算RQ1指标的Python Jupyter Notebook。如该Notebook中所述，使用时需修改首个代码块中包含的变量值。 - independent-prompting：包含与独立式提示相关的数据集，每行包含以下字段： - Link：待标注工件的链接 - Prompt：发送至GPT-4-Turbo的提示词 - FinalTag：复现研究中的工件编码结果 - chatgpt_output_text：GPT-4-Turbo的输出文本 - chatgpt_output：从GPT-4-Turbo输出中解析得到的编码结果 - Author1：第一位标注者对编码结果的评估 - Author2：第二位标注者对编码结果的评估 - FinalOutput：整合后的评估结果 - incremental-prompting：包含与增量式提示相关的数据集（格式与独立式提示数据集一致） - results：包含RQ1定量结果文件，文件命名格式为RQ1_<<Dataset>>_<<Prompt method>>_<<ExcludingNegative>>_<<MetricAggregation>>.csv，其中： - Dataset为数据集名称， - Prompt method表示结果对应独立式还是增量式提示方法， - Excluding Negatives（针对适用该设置的数据集）表示是否通过剔除负样本得到结果， - MetricAggregation（若适用）表示指标的聚合方式（宏平均或加权平均）。 上述文件包含以下列：Dataset、匹配类型、准确率（Accuracy）、精确率（Precision）、召回率（Recall）、F1值（F1 Score）与科恩卡帕系数(Cohen's Kappa)。 RQ2：包含用于开展RQ2分析的脚本、所用数据集与输出结果。 RQ2_SetStats.ipynb为用于执行分析的Python Jupyter Notebook。该脚本的输入文件包含以下几类（均存放于对应目录中）： 1. RQ1数据文件：RQ1_DLFaults_Issues.csv、RQ1_DLFaults_Commits.csv与RQ1_DLFaults_SO.csv，已合并为单个.csv文件RQ1_DLFaults.csv。这些文件与RQ1中所用的文件一致。 2. 映射文件：RQ2_Mappings_DRL.csv、RQ2_Mappings_Functional.csv、RQ2_Mappings_DLFaults.csv。这些文件包含人工标注标签（HumanTags）、GPT-4-Turbo标签（Final Tags）以及匹配类型（MatchType）的映射关系。 3. 整合过程中新增的标注代码：RQ2_newCodes_DRL.csv、RQ2_newCodes_Functional.csv、RQ2_newCodes_DLFaults.csv，标注信息包括：新代码、旧代码、人工标注代码、匹配类型。 4. 集合文件：RQ2_Sets_DRL.csv、RQ2_Sets_Functional.csv、RQ2_Sets_DLFaults.csv。每个文件包含以下列： - HumanTags：原始数据集中的标签列表 - InitialTags：RQ1中的标签集合 - ConsolidatedTags：已完成整合的标签 - FinalTags：最终标签集合（RQ2的结果，用于RQ3） - NewTags：整合过程中创建的新标签 RQ2_Set_Metrics.csv：报告了RQ2的输出指标，包括精确率、召回率、F1值与杰卡德系数(Jaccard)。