Spark Dataset: Fine-Grained Code Changes (Diff-Level) Linked to Issue Descriptions for Software Traceability Analysis
收藏资源简介：
Dataset: Apache Spark Commit–Task TraceabilityProject: apache/spark | Organization: Apache Software Foundation | Language: Scala, Java, Python (.scala .java .py) 1. CONTEXT AND MOTIVATIONThis dataset supports research into traceability between task descriptions and code changes inlarge-scale distributed data processing. Apache Spark is one of the most widely adopted big-dataframeworks, with contributions from hundreds of organizations worldwide. Its Apache Jira trackeruses structured ticket identifiers (SPARK-NNNN) that are consistently referenced in commitmessages, yielding a high-quality linked dataset. The multi-language codebase (Scala, Java,Python) makes it particularly interesting for studying language-specific change patterns. Commitsspan over 15 years of active development. 2. COLLECTION METHODOLOGYSource: GitHub repository — https://github.com/apache/sparkCommits: Extracted via the GitHub API (all branches, full history)Task linking: Jira ticket references extracted from commit messages using the pattern SPARK-[0-9]+ ; each reference was resolved via the Apache Jira APIIssue content: Title, description, and comment thread fetched via Apache Jira REST API (https://issues.apache.org/jira/projects/SPARK) for each referenced ticketTime range: 2010-03-29 to 2025-10-02Anonymization: Author names and e-mail addresses replaced with sequential pseudonyms (User1, User2, …) prior to publication 3. DATASET STRUCTURETable COMMITS — one row per commit ID INTEGER Primary key SHA TEXT Full commit hash AUTHOR_NAME TEXT Anonymized author pseudonym (e.g. User42) AUTHOR_EMAIL TEXT Anonymized e-mail (e.g. user42@anon.local) CMT_DATE TEXT Commit timestamp (ISO-8601 with timezone) MESSAGE BLOB Full commit message text PATH BLOB List of file paths changed in this commit DIFF BLOB Unified diff of the commit TASK_NAME TEXT Linked Jira ticket ID, e.g. SPARK-12345 (NULL if no link detected) Table TASK — one row per unique linked Jira ticket ID INTEGER Primary key (autoincrement) NAME TEXT Jira ticket ID (matches TASK_NAME in COMMITS) TITLE TEXT Issue summary as fetched from Jira DESCRIPTION TEXT Issue description body COMMENTS TEXT Serialized comment thread 4. BASIC STATISTICSTotal commits: 306,804Commits with linked task: 210,803 (68.7 %)Commits without linked task: 96,001 (31.3 %)Unique linked issues: 30,478Issues with description text: 22,763 (74.7 %)Unique authors (anonymized): 3,509Date range: 2010-03-29 — 2025-10-02
数据集：Apache Spark提交-任务可追溯性 项目：apache/spark | 所属机构：Apache软件基金会 | 开发语言：Scala、Java、Python（文件后缀为.scala、.java、.py） 1. 研究背景与动因 本数据集用于支撑大规模分布式数据处理场景下任务描述与代码变更之间可追溯性的相关研究。Apache Spark是当前应用最广泛的大数据框架之一，全球有数百家机构为其贡献代码。其Apache Jira追踪系统采用结构化的工单标识符（SPARK-NNNN），且提交信息中会统一引用此类标识符，由此可构建高质量的关联数据集。该多语言代码库（Scala、Java、Python）为研究特定语言的代码变更模式提供了优质研究场景，且覆盖了该项目15年的活跃开发历史。 2. 数据采集方法 数据来源：GitHub 仓库 — https://github.com/apache/spark 提交数据：通过GitHub API提取（包含所有分支及完整提交历史） 任务关联：通过匹配模式`SPARK-[0-9]+`从提交信息中提取Jira工单引用，并通过Apache Jira API解析每个引用对应的工单信息 工单内容：通过Apache Jira REST API（https://issues.apache.org/jira/projects/SPARK）获取每个关联工单的标题、描述及评论线程 时间范围：2010-03-29 至 2025-10-02 匿名化处理：发布前将提交作者的姓名与电子邮箱替换为顺序化名（如User1、User2……） 3. 数据集结构 表 COMMITS — 每条记录对应一次提交 | 字段名 | 数据类型 | 说明 | |----------------|----------|----------------------------------------------------------------------| | ID | INTEGER | 主键 | | SHA | TEXT | 完整提交哈希值 | | AUTHOR_NAME | TEXT | 匿名化后的作者化名（例如User42） | | AUTHOR_EMAIL | TEXT | 匿名化后的电子邮箱（例如user42@anon.local） | | CMT_DATE | TEXT | 提交时间戳（采用带时区的ISO-8601格式） | | MESSAGE | BLOB | 完整提交信息文本 | | PATH | BLOB | 本次提交变更的文件路径列表 | | DIFF | BLOB | 本次提交的统一差异内容 | | TASK_NAME | TEXT | 关联的Jira工单ID，例如SPARK-12345（未检测到关联时为NULL） | 表 TASK — 每条记录对应一个唯一关联的Jira工单 | 字段名 | 数据类型 | 说明 | |----------------|----------|----------------------------------------------------------------------| | ID | INTEGER | 主键（自增） | | NAME | TEXT | Jira工单ID（与COMMITS表中的TASK_NAME字段匹配） | | TITLE | TEXT | 从Jira获取的工单摘要 | | DESCRIPTION | TEXT | 工单描述正文 | | COMMENTS | TEXT | 序列化后的评论线程 | 4. 基础统计指标 总提交次数：306,804次 关联了任务的提交次数：210,803次（占比68.7%） 未关联任务的提交次数：96,001次（占比31.3%） 唯一关联工单数量：30,478个 带有描述文本的工单数量：22,763个（占比74.7%） 匿名化后的唯一作者数量：3,509位 时间范围：2010-03-29 — 2025-10-02
