Artifact of the paper "Why Are AI Agent–Involved Pull Requests (Fix-Related) Remain Unmerged? An Empirical Study"
收藏资源简介:
This replication package provides all datasets, scripts, and analysis results required to reproduce the findings reported in “Why Are AI Agent–Involved Pull Requests (Fix-Related) Remain Unmerged? An Empirical Study.” The package is designed to support transparency, reproducibility, and independent verification of both the quantitative and qualitative analyses conducted in this study. The dataset folder includes the AIDEV-POP dataset, which contains 33,596 AI-authored pull requests mined from GitHub, as well as all derived subsets used in the paper. From this dataset, we extracted 8,106 fix-related pull requests authored by five AI coding agents (OpenAI Codex, GitHub Copilot, Devin, Cursor, and Claude Code). The scripts folder contains all programs used to preprocess the data, conduct the quantitative and qualitative analyses, and generate the results reported in the paper. These scripts implement the full analysis pipeline, including identification of fix-related pull requests, outcome classification (merged, closed without merging, open), agent-wise aggregation, merge-time computation, and analysis of code review activity, as well as a visualization chart for failure topic distribution. The results folder contains the outputs generated from the analyses conducted in this study. For RQ1, it includes the distribution of pull request merge times, provided as density and boxplot figures. For RQ2, it contains the spreadsheet documenting the failure analysis of fix-related pull requests that were closed without merging. These artifacts correspond directly to the results reported in the paper and support verification of the empirical findings.
本复现包包含了复现《为何涉及AI智能体(AI Agent)的修复相关拉取请求(Pull Request,PR)始终无法合并?一项实证研究》一文所报告的研究发现所需的全部数据集、脚本与分析结果。本包旨在提升本研究中定量与定性分析的透明度、可复现性与独立验证性。 数据集文件夹包含AIDEV-POP数据集,该数据集收录了从GitHub中挖掘得到的33596个AI生成拉取请求,以及论文中使用的全部衍生子集。本数据集从中提取出了由5种AI编码智能体(OpenAI Codex、GitHub Copilot、Devin、Cursor及Claude Code)生成的8106个修复相关拉取请求。 脚本文件夹包含了用于数据预处理、开展定量与定性分析以及生成论文中报告结果的全部程序。这些脚本实现了完整的分析流程,涵盖修复相关拉取请求的识别、结果分类(已合并、未合并关闭、仍处于开放状态)、按智能体聚合、合并时长计算、代码审查活动分析,以及用于展示失败主题分布的可视化图表。 结果文件夹包含本研究中分析生成的全部输出产物。针对研究问题1(RQ1),其包含拉取请求合并时长的分布数据,以密度图与箱线图形式提供。针对研究问题2(RQ2),其包含记录未合并关闭的修复相关拉取请求失败分析情况的电子表格。这些产出与论文中报告的研究结果直接对应,可用于验证本实证研究的发现。



