Replication package: The Prevalence of Code Review Guidelines for GUI-Based Testing in Open-Source

Replication package for the study "The Prevalence of Code Review Guidelines for GUI-Based Testing in Open-Source" Summary of files included: results.xls contains the final results of mapping code review guidelines to observed code review comments in pull requests. repositories_all.zip includes a CSV file listing all identified GitHub repositories that meet our search criteria. repositories-top100.xlsx provides an overview of the top 100 (by star count) repositories that are considered in the study. GitHub-Crawler-code-only includes the scripts used to gather all relevant data from GitHub. GitHub-Crawler-with-data includes the scripts used to gather all relevant data from GitHub, with intermediate results from the crawling process, such as all pull requests and metadata for each repository. (~3.6 GB uncompressed) CodeBook.xlsx provides a codebook that defines the guidelines and the rules for mapping comments to these guidelines. Coding_with_IRR_metrics.zip contains the coding of a random sample of 200 PRs by an independent coder, including inter-rater reliability (IRR) metrics. How to Read the Results The file results.xls contains all pull requests (PRs) from 84 repositories that modify GUI-based test files. The columns in the main sheet are as follows: Repository: The name of the GitHub repository. PR_id: The unique identifier for a PR within a repository. An ID may appear in multiple repositories as it is a sequential number assigned in each repository. URL: A link that allows direct access to the PR on GitHub. Guidelines: Our mapping of code review comments to previously proposed guidelines. If multiple guidelines are identified in a PR, they are separated by the pipe symbol "|". Notes: Researcher notes regarding the mapping of comments to guidelines. These notes are used to highlight specific aspects of the PR that led to the mapping to a guideline. Checked: The letter "y" indicates the PR was reviewed by both authors who performed the mapping. A "d" marks PRs that contain only emojis, which we do not consider as a code review. Comments on File Changes (Threads): Code review comments that directly address file changes (also known as threads in GitHub). These are the comments used for the mapping to guidelines. Each thread (comments on file changes) includes a path to the changed file, comments from reviewers, and a horizontal line indicating the end of the thread (as a visual indicator for analysis). Author of PR: The GitHub user who created the PR. This is relevant to determine whether a code review comment was made by the PR author, for example, to highlight a specific part of the change. General PR Comments: General comments about the PR that do not target specific file changes. These comments were not used for the mapping to guidelines, but are included in this spreadsheet for additional context. The other sheets in the results.xls file are: Stats: An overview of the number of PRs that were analyzed by two authors. Rules: These are guidelines on how to map specific comments. The rules were developed through discussions between the two authors who performed the mapping and were documented to ensure consistent mapping throughout the various sessions. These rules were also used to create the first version of the codebook, which is included in the CodeBook.xlsx file. Sessions: Protocols for the sessions conducted for mapping guidelines by the two authors. G5.6: This section represents different concerns along with the number of PRs in which each concern appears. It was used to further subdivide guideline G5.6 into more specialized guidelines. To analyze or replicate the results, the columns "Guidelines", "Comments on File Changes (Threads)", and "Author of PR" are particularly important. The "Guidelines" column indicates which code review guidelines were identified in the PR, while the "Comments on File Changes (Threads)" column provides the specific comments that led to the mapping to these guidelines. The formatting of suggested changes in the "Comments on File Changes (Threads)" is not always easy to understand, especially for larger changes or minor formatting changes. In unclear cases, the URL column can be used to access the PR on GitHub and review the comments in their original formatting. The "Author of PR" column should be used to determine whether the PR author or a reviewer makes a comment. If the PR author does the first comment of a thread, it is to highlight specific file changes for the reviewer (G3.9). Coding of Guidelines If you want to perform coding (a qualitative data analysis strategy) on code review comments to map them to guidelines (codes), you can use the codebook provided in CodeBook.xlsx. This codebook defines the guidelines and provides rules for mapping comments to these guidelines. In particular, it provides: a description of each guideline (code) explains when to use a code explains when not to use a code provides examples of comments for the "when to use" case All four versions of the codebook and a changelog are included in the spreadsheet, which shows how the codebook has been improved over time.本研究为论文《开源项目中基于GUI的测试代码评审指南的普及现状》的复现数据包。 所含文件概述： results.xls：收录了将代码评审指南映射至拉取请求（Pull Request，以下简称PR）中观测到的代码评审评论的最终结果。 repositories_all.zip：内含一个CSV文件，罗列了所有符合本研究检索条件的已识别GitHub仓库。 repositories-top100.xlsx：展示了本研究纳入的前100个（按星标数量排序）GitHub仓库的概况。 GitHub-Crawler-code-only：收录了用于从GitHub抓取所有相关数据的脚本文件。 GitHub-Crawler-with-data：收录了用于从GitHub抓取所有相关数据的脚本，附带抓取过程产生的中间结果，例如所有PR及每个仓库的元数据。（解压后约3.6 GB） CodeBook.xlsx：提供了一份编码手册（codebook），用于定义本研究的评审指南以及将评论映射至对应指南的规则。 Coding_with_IRR_metrics.zip：收录了由独立编码员对200个随机抽样PR进行的编码结果，包含编码者间信度（inter-rater reliability，IRR）指标。 结果文件使用说明 results.xls 文件收录了来自84个修改了基于图形用户界面（Graphical User Interface，以下简称GUI）的测试文件的GitHub仓库的全部PR。主工作表的列信息如下： Repository（仓库名称）：GitHub仓库的名称。 PR_id：仓库内PR的唯一标识符。由于每个仓库内的ID均为按顺序分配的序列号，因此同一ID可能会出现在多个仓库中。 URL：可直接访问GitHub上对应PR的链接。 Guidelines（评审指南映射）：本研究将代码评审评论映射至预设指南的结果。若一个PR中识别出多条指南，则以竖线符号"|"分隔。 Notes（研究备注）：关于评论与指南映射关系的研究者备注，用于说明导致该映射结果的PR特定细节。 Checked（审核标记）：标记"y"表示该PR已由两名参与映射工作的研究者共同审核；标记"d"表示该PR仅包含表情符号，本研究不将其视为有效代码评审。 Comments on File Changes (Threads)（文件变更评论线程）：直接针对文件变更的代码评审评论（GitHub中亦称为线程），即用于映射至评审指南的评论。每个线程（文件变更评论）包含变更文件路径、评审员的评论，以及用于标识线程结束的水平线（用于分析时的视觉标记）。 Author of PR（PR创建者）：创建该PR的GitHub用户。该字段可用于判断某条代码评审评论是否由PR创建者发出，例如用于标注变更的特定部分。 General PR Comments（PR通用评论）：针对该PR的通用评论，不指向特定文件变更。此类评论未被用于映射至评审指南，但收录于此表以提供额外上下文信息。 results.xls 文件的其他工作表如下： Stats（统计信息）：由两名研究者共同审核的PR数量概况。 Rules（映射规则）：关于如何映射特定评论的指南。本规则由两名参与映射工作的研究者通过讨论制定并记录，以确保在各轮映射过程中保持一致性。本规则亦用于创建CodeBook.xlsx文件中收录的首版编码手册。 Sessions（工作会话记录）：两名研究者开展评审指南映射工作的会话记录文档。 G5.6：本部分罗列了各类关注点及其对应的PR数量，用于将指南G5.6进一步细分为更细分的专项指南。 若需分析或复现本研究结果，"Guidelines"、"Comments on File Changes (Threads)"及"Author of PR"三列尤为关键。其中，"Guidelines"列指明了PR中识别出的代码评审指南；"Comments on File Changes (Threads)"列则提供了用于映射至对应指南的具体评论内容。"Comments on File Changes (Threads)"列中变更建议的格式有时难以解读，尤其是针对大规模变更或细微格式调整的情况。若存在模糊之处，可通过URL列直接访问GitHub上的PR，以查看评论的原始格式。"Author of PR"列可用于判断评论是由PR创建者还是评审员发出：若PR创建者为某线程的首条评论发布者，则其目的通常是向评审员标注变更的特定部分（对应指南G3.9）。 评审指南编码 若需对代码评审评论执行编码（一种质性数据分析策略）以将其映射至评审指南（编码项），可使用CodeBook.xlsx中提供的编码手册。该手册定义了本研究的评审指南，并提供了将评论映射至对应指南的规则。 具体而言，该手册包含以下内容： - 每条指南（编码项）的描述 - 编码项的适用场景 - 编码项的禁用场景 - "适用场景"下的评论示例 本工作表中还收录了该编码手册的全部四个版本及一份变更日志，用于展示该手册随时间推移的迭代优化过程。