Spark Dataset: Fine-Grained Code Changes (Diff-Level) Linked to Issue Descriptions for Software Traceability Analysis
收藏资源简介：
Dataset: Apache Spark Commit–Task TraceabilityProject: apache/spark | Organization: Apache Software Foundation | Language: Scala, Java, Python (.scala .java .py) 1. CONTEXT AND MOTIVATIONThis dataset supports research into traceability between task descriptions and code changes inlarge-scale distributed data processing. Apache Spark is one of the most widely adopted big-dataframeworks, with contributions from hundreds of organizations worldwide. Its Apache Jira trackeruses structured ticket identifiers (SPARK-NNNN) that are consistently referenced in commitmessages, yielding a high-quality linked dataset. The multi-language codebase (Scala, Java,Python) makes it particularly interesting for studying language-specific change patterns. Commitsspan over 15 years of active development. 2. COLLECTION METHODOLOGYSource: GitHub repository — https://github.com/apache/sparkCommits: Extracted via the GitHub API (all branches, full history)Task linking: Jira ticket references extracted from commit messages using the pattern SPARK-[0-9]+ ; each reference was resolved via the Apache Jira APIIssue content: Title, description, and comment thread fetched via Apache Jira REST API (https://issues.apache.org/jira/projects/SPARK) for each referenced ticketTime range: 2010-03-29 to 2025-10-02Anonymization: Author names and e-mail addresses replaced with sequential pseudonyms (User1, User2, …) prior to publication 3. DATASET STRUCTURETable COMMITS — one row per commit ID INTEGER Primary key SHA TEXT Full commit hash AUTHOR_NAME TEXT Anonymized author pseudonym (e.g. User42) AUTHOR_EMAIL TEXT Anonymized e-mail (e.g. user42@anon.local) CMT_DATE TEXT Commit timestamp (ISO-8601 with timezone) MESSAGE BLOB Full commit message text PATH BLOB List of file paths changed in this commit DIFF BLOB Unified diff of the commit TASK_NAME TEXT Linked Jira ticket ID, e.g. SPARK-12345 (NULL if no link detected) Table TASK — one row per unique linked Jira ticket ID INTEGER Primary key (autoincrement) NAME TEXT Jira ticket ID (matches TASK_NAME in COMMITS) TITLE TEXT Issue summary as fetched from Jira DESCRIPTION TEXT Issue description body COMMENTS TEXT Serialized comment thread 4. BASIC STATISTICSTotal commits: 306,804Commits with linked task: 210,803 (68.7 %)Commits without linked task: 96,001 (31.3 %)Unique linked issues: 30,478Issues with description text: 22,763 (74.7 %)Unique authors (anonymized): 3,509Date range: 2010-03-29 — 2025-10-02
