Online Appendix for Show Your Title! A Scoping Review on Verbalization in Software Engineering with LLM-Assisted Screening

Validation & Replication Appendix This appendix accompanies "Show Your Title! A Scoping Review on Verbalization in Software Engineering with LLM-Assisted Screening" and contains all artifacts and data necessary to replicate and validate our scoping review methodology. Keywords: verbalization-based techniques · human-centered SE · cognitive aspects of SE · meta-research · qualitative research in SE · knowledge representation in SE · software development practices Overview & Validation Focus This appendix serves a dual validation purpose: enabling complete methodological replication while demonstrating the reliability and consistency of our LLM-assisted approach. All files support our claims about GPT's 91%-92% internal consistency and acceptable 13% disagreement rate between GPT and human reviewers. File Descriptions Keyword Files (cf. se_keywords.json and psy_keywords.json)- `se_keywords.json` - Software Engineering keywords organized by inclusion question tags - `psy_keywords.json` - Psychology keywords organized by inclusion question tags Purpose: Complete list of keywords used for literature search. Essential for replicating our search strategy that operationalizes the seven inclusion questions. Venue Information (cf. key venues.md)- `key venues.md` - Selected venues per field used in our literature search Purpose: Documents the publication venues that define disciplinary boundaries for our intersection analysis. GPT Processing Materials (cf. multi paper relevancy prompt for OpenAI.md and asses_relevancy_with_gpt_by_title.py)- `multi paper relevancy prompt for OpenAI.md` - Standardized prompt used for GPT-4.1 relevance assessment- `asses_relevancy_with_gpt_by_title.py` - Python script for conducting batch relevance assessments using GPT Purpose: Enable exact replication of our LLM-assisted screening methodology using title-based assessment in batches of 10 papers. Results and Analysis Files (cf. relevant_justification_grouping_by_GPT_runX.json and tags_distribution.json)- `relevant_justification_grouping_by_GPT_runX.json` (where X = 1, 2, or 3) - GPT-generated thematic grouping results across three independent runs, each containing: - `label`: Theme name - `reasoning_pattern`: Informal theme definition - `prominence_reason`: Explanation of theme prominence - `justifications`: Array of example justifications - `tags_distribution.json` - Statistical distribution of inclusion question tags across relevant papers, supporting Figure 2 visualization Purpose: Support our thematic analysis and enable replication of the prominent theme identification process. Validation Data (cf. manual_validation_assessment_agreement.json and psy_on_se\GPT_relevancy_assessment_agreement.json)- `manual_validation_assessment_agreement.json` - Human reviewer agreement data showing perfect agreement achieved in 60% of cases, with 40% showing partial disagreements - `psy_on_se\GPT_relevancy_assessment_agreement.json` and `se_on_py\GPT_relevancy_assessment_agreement.json` - GPT internal consistency data demonstrating 91%-92% self-agreement across multiple assessments Purpose: Validate our claims about GPT consistency exceeding human reviewer agreement and support the 13% final disagreement rate between GPT and human judgments. Scripts Note that some scripts might need some minor editing, so the included paths match your file system.Some of the data were manually aggregated by running the same scripts multiply times.- src - all the scripts we used to produce the above mentioned raw data- pyproject.toml - project metadata, this files makes it easier to install the dependencies Key Statistics Supported by This Data - 9,265 papers processed after deduplication (5,386 SE on PSY + 3,879 PSY on SE)- 1,675 papers marked as relevant through majority voting across three GPT runs- 792 papers (14.7%) marked relevant in SE on PSY intersection - 883 papers (22.76%) marked relevant in PSY on SE intersection- 100-paper validation sample with 95% confidence level and margin of error below 10%- Sample composition: 42% PSY on SE papers, 58% SE on PSY papers- Relevance distribution in sample: PSY on SE (23% relevant, 77% irrelevant), SE on PSY (20% relevant, 80% irrelevant) Validation Results - GPT internal consistency: 91%-92% perfect self-agreement - Human reviewer agreement: 60% perfect agreement, 40% partial disagreements- GPT vs Human disagreement: 13% final disagreement rate after consolidation- GPT divergence rates: 9%-8% partial divergence for SE on PSY and PSY on SE respectively Technical Requirements for Replication - Scopus API access for literature search replication- GPT-4.1 (2025-05-15) for exact prompt execution- Python environment for script execution- JSON processing capabilities for data analysis This appendix enables full replication of our methodology while validating the effectiveness of LLM-assisted screening in interdisciplinary scoping reviews.验证与复现附录 本附录配套论文《亮出你的标题！基于大语言模型（Large Language Model, LLM）辅助筛选的软件工程语言化研究范围综述》(Show Your Title! A Scoping Review on Verbalization in Software Engineering with LLM-Assisted Screening)，包含复现与验证本研究范围综述方法论所需的全部科研产出与数据集。 关键词：基于语言化的技术 · 以人为中心的软件工程 · 软件工程认知维度 · 元研究 · 软件工程定性研究 · 软件工程知识表示 · 软件开发实践 概述与验证目标 本附录兼具双重验证功能：既可支撑方法论的完整复现，也可论证大语言模型辅助研究方法的可靠性与一致性。所有配套文件均支撑我们的两项核心论断：GPT（Generative Pre-trained Transformer）内部一致性达91%~92%，且GPT与人工评审者间的可接受分歧率为13%。 文件说明 关键词文件 (参见se_keywords.json与psy_keywords.json) - `se_keywords.json`：按纳入问题标签组织的软件工程关键词列表 - `psy_keywords.json`：按纳入问题标签组织的心理学关键词列表 用途：本清单为文献检索所用的完整关键词集合，是复现我们针对七个纳入问题制定的检索策略的核心依据。 出版源信息 (参见key venues.md) - `key venues.md`：本研究文献检索所用的各领域精选学术出版源 用途：本文件记录了为交叉分析划定学科边界的学术出版源。 GPT相关处理材料 (参见multi paper relevancy prompt for OpenAI.md与asses_relevancy_with_gpt_by_title.py) - `multi paper relevancy prompt for OpenAI.md`：用于GPT-4.1相关性评估的标准化提示词 - `asses_relevancy_with_gpt_by_title.py`：用于批量开展基于标题的GPT相关性评估的Python脚本 用途：支持通过每批次10篇论文的标题评估，完整复现本研究的大语言模型辅助筛选方法论。 结果与分析文件 (参见relevant_justification_grouping_by_GPT_runX.json与tags_distribution.json，其中X=1、2或3) - `relevant_justification_grouping_by_GPT_runX.json`：三次独立运行下GPT生成的主题分组结果，每份文件包含以下字段： - `label`：主题名称 - `reasoning_pattern`：主题的非正式定义 - `prominence_reason`：主题显著性的解释说明 - `justifications`：示例理由数组 - `tags_distribution.json`：纳入文献的纳入问题标签统计分布，用于支撑图2的可视化呈现 用途：支撑本研究的主题分析，并支持复现核心主题识别流程。 验证数据集 (参见manual_validation_assessment_agreement.json与psy_on_seGPT_relevancy_assessment_agreement.json) - `manual_validation_assessment_agreement.json`：人工评审一致性数据，显示60%的案例达成完全一致，40%存在部分分歧 - `psy_on_seGPT_relevancy_assessment_agreement.json`与`se_on_pyGPT_relevancy_assessment_agreement.json`：GPT内部一致性数据，证明多次评估下GPT的自我一致性达91%~92% 用途：验证我们关于GPT一致性优于人工评审一致性的论断，并支撑GPT与人工判断间13%的最终分歧率。 脚本文件 > 注：部分脚本需进行少量编辑，以确保内置路径与本地文件系统匹配。部分数据集通过多次运行同一脚本手动聚合得到。 > - `src`：用于生成上述原始数据的全部脚本 > - `pyproject.toml`：项目元数据文件，用于简化依赖安装流程 本数据支撑的核心统计量 - 去重后共处理9265篇文献（其中“软件工程聚焦心理学”类5386篇，“心理学聚焦软件工程”类3879篇） - 经三次GPT运行的多数投票后，共筛选出1675篇相关文献 - 792篇（14.7%）为“软件工程聚焦心理学”交叉领域的相关文献 - 883篇（22.76%）为“心理学聚焦软件工程”交叉领域的相关文献 - 100篇文献的验证样本，置信水平达95%，边际误差低于10% - 样本构成：42%为“心理学聚焦软件工程”类文献，58%为“软件工程聚焦心理学”类文献 - 样本内相关性分布：“心理学聚焦软件工程”类（23%相关，77%不相关），“软件工程聚焦心理学”类（20%相关，80%不相关） 验证结果 - GPT内部一致性：91%~92%的完全自我一致性 - 人工评审一致性：60%达成完全一致，40%存在部分分歧 - GPT与人工评审分歧率：整合后最终分歧率为13% - GPT分歧率：“软件工程聚焦心理学”与“心理学聚焦软件工程”类文献的部分分歧率分别为9%与8% 复现所需技术条件 - Scopus API访问权限：用于复现文献检索流程 - GPT-4.1（2025-05-15）：用于精确执行预设提示词 - Python运行环境：用于执行配套脚本 - JSON数据处理能力：用于数据分析 本附录可实现本研究方法论的完整复现，同时验证大语言模型辅助筛选在跨学科范围综述中的有效性。