Replication Package: LLM-Generated Threats to Validity for ICSE Papers

This replication package contains the data and scripts used to generate and evaluate LLM-generated threats to validity for 375 ICSE research papers. Contents: Input Papers (Input_Papers.zip) - The dataset of 375 ICSE papers used as input for threat generation. Threat Generation Prompt - The prompt used to generate threats to validity for all 375 papers. Each paper was passed (excluding its existing threats to validity section) to generate_threats.ipynb alongside this prompt. Generated Threats Dataset (threat_dataset.csv) - The LLM-generated threats for all 375 papers. Note: The dataset contains 110 entries rather than 375 because the generation script occasionally combines multiple papers into a single row when Gemini produces a low number of threats for individual papers. Scripts: paper_split.py: Splits papers (separating the TTV section) for processing. generate_threats.ipynb: Notebook for running the threat generation across all papers. generate_rubric.py: LLM-as-judge script that grades generated threats against a 4-criterion rubric (Relevance, Specificity, Clarity, Mitigation). rubric_count.py: Counts how many threats fall into each scoring bucket per criterion (e.g., how many threats had High Specificity, how many had No Impact for Relevance). compare_threats.py: Semantically compares LLM-generated threats against the paper's actual TTV section using Gemini 2.5 Flash. Buckets each threat into matched, only_AI, or only_Paper across the three validity categories (external, internal, construct). matched threats appear in both the prediction and the paper, only_AI threats were predicted but not in the paper, and only_Paper threats were in the paper but missed by the AI. Scoring Sample (scoring.csv) - A sample of manually graded threats used to establish inter-rater agreement for the rubric criteria.本复现包包含用于生成与评估针对375篇ICSE会议研究论文的大语言模型（Large Language Model，以下简称LLM）生成的有效性威胁所需的数据及脚本。 数据集内容： Input Papers (Input_Papers.zip)—— 作为威胁生成输入的375篇ICSE会议论文数据集。 Threat Generation Prompt—— 用于为全部375篇论文生成有效性威胁的提示词。每篇论文（剔除其原有的有效性威胁章节）将与该提示词一同传入generate_threats.ipynb脚本。 Generated Threats Dataset (threat_dataset.csv)—— 针对全部375篇论文由LLM生成的有效性威胁。注意：本数据集仅包含110条条目而非375条，原因是当Gemini为单篇论文生成的威胁数量较少时，生成脚本会偶尔将多篇论文合并至同一行中。 脚本： paper_split.py：用于拆分论文（分离其中的有效性威胁（Threats to Validity，下文简称TTV）章节）以进行后续处理的脚本。 generate_threats.ipynb：用于对所有论文执行威胁生成的Jupyter Notebook脚本。 generate_rubric.py：以LLM作为评判者的脚本，用于依据四项评分标准（相关性、特异性、清晰度、缓解性）对生成的威胁进行评分。 rubric_count.py：统计各评分维度下落入各评分区间的威胁数量（例如，获得高特异性评分的威胁数量、相关性评分为无影响的威胁数量）。 compare_threats.py：使用Gemini 2.5 Flash对LLM生成的威胁与论文原有的TTV章节进行语义对比，将每条威胁按三类有效性范畴（外部有效性、内部有效性、建构有效性）划分为匹配项、仅AI生成项（only_AI）或仅论文存在项（only_Paper）。其中，匹配项指同时出现在模型预测与论文原文中的威胁；仅AI生成项指仅被模型预测但未出现在论文原文中的威胁；仅论文存在项指出现在论文原文中但未被模型识别的威胁。 评分样本（scoring.csv）—— 用于为该评分标准建立评分者间一致性的手动标注威胁样本。