A Dataset of Contributor Activities in the NumFocus Open-Source Community

The NumFocus dataset provides a comprehensive representation of contributor activity across 58 open-source projects supported by the NumFocus organization. Spanning a three-year observation period (January 2022 to December 2024), this dataset captures the dynamics of open-source collaboration within a defined community of scientific and data-driven software projects.

To address the challenges of interpreting raw GitHub event logs, the dataset introduces two structured levels of abstraction: actions and activities. Actions offer a detailed view of individual operations, such as creating branches or pushing commits, while activities aggregate related actions into high-level tasks, such as merging pull requests or resolving issues. This hierarchy bridges the gap between granular operations and contributors’ broader intentions.

The primary dataset focuses on activities, providing a high-level overview of contributor behavior. For users requiring more granular analysis, a complementary dataset of actions is also included.

The dataset is accompanied by a Python-based command-line tool that automates the transformation of raw GitHub event logs into structured actions and activities. The tool, along with its configurable mapping files and scripts, is publicly available at ghmap.

The dataset is distributed across the following files:

NumFocus_Jan22-Dec24_GH_Actions.zip: Contains 2,716,910 actions in JSON Lines format, capturing individual contributor operations.

NumFocus_Jan22-Dec24_GH_Activities.zip: Contains 2,278,299 activities in JSON Lines format, representing high-level tasks derived from grouped actions.

action_schema.json: A validation schema in JSON format to ensure consistency in interpreting the actions dataset.

activity_schema.json: A validation schema for validating and integrating the activities dataset.

Actions: Low-Level Operations

Actions represent the most granular level of recorded operations, derived from raw GitHub events. Each action corresponds to a single, well-defined contributor operation, such as pushing code to a repository, opening a pull request, or commenting on an issue. This level preserves the technical details necessary for tracing individual operations while standardizing event data to facilitate analysis across repositories.

Each action record captures a single contributor operation and includes the following attributes:

action: Specifies the type of operation (e.g., PushCommits, OpenPullRequest, or CreateBranch).

event_id: A unique identifier linking the action to its originating GitHub event.

date: The timestamp of the action, recorded in ISO 8601 format.

actor: Contains details about the contributor performing the action, including a persistent id and their GitHub login.

repository: Provides information about the repository where the action occurred, including its id, name, and associated organisation.

details: Stores additional attributes specific to the action type, extracted from the payload of the corresponding GitHub event (e.g., for a PushCommits action, the details include the branch reference and the number of commits; for an OpenPullRequest action, the details include the pull request’s title, labels, state, and creation and update dates).

The dataset encompasses 24 distinct action types, each derived from specific GitHub events and representing a well-defined contributor operation:

AddMember: Tracks the addition of a new collaborator to a repository.

CloseIssue: Indicates that an issue has been marked as closed by a contributor.

ClosePullRequest: Represents the closure of a pull request without merging its changes.

CommentCommit: Captures comments made directly on specific commits within a repository.

CreateBranch: Logs the creation of a new branch within a repository.

CreateIssueComment: Tracks comments added to existing issues.

CreatePullRequestComment: Records comments made on pull requests, including discussions on the changes proposed.

CreatePullRequestReview: Represents the submission of a review for a pull request.

CreatePullRequestReviewComment: Captures inline comments added during a pull request review process.

CreateRepository: Represents the creation of a new GitHub repository.

CreateTag: Logs the creation of a tag, often associated with versioning or releases.

DeleteBranch: Indicates that an existing branch has been deleted from a repository.

DeleteTag: Tracks the deletion of a tag within a repository.

ForkRepository: Captures the action of forking a repository to create a copy under a different account.

MakeRepositoryPublic: Represents the change of a private repository’s visibility to public.

ManageWikiPage: Logs edits or updates made to a repository’s wiki pages.

MergePullRequest: Indicates that a pull request has been merged, integrating its changes into the base branch.

OpenIssue: Captures the creation of a new issue within a repository.

OpenPullRequest: Represents the initiation of a new pull request to propose changes.

PublishRelease: Tracks the publication of a release, often tied to specific tags and associated metadata.

PushCommits: Records push events, detailing branches and commits included in the operation.

ReopenIssue: Indicates that a previously closed issue has been reopened for further action.

ReopenPullRequest: Captures the reopening of a previously closed pull request.

StarRepository: Tracks when a user stars a repository to bookmark it or show support.

Example of action record:

{
"action":"CloseIssue",
"event_id":"26170139709",
"date":"2023-01-01T20:19:58Z",
"actor":{
"id":1282691,
"login":"KristofferC"
},
"repository":{
"id":1644196,
"name":"JuliaLang/julia",
"organisation":"JuliaLang",
"organisation_id":743164
},
"details":{
"issue":{
"id":1515182791,
"number":48062,
"title":"Bad default number of BLAS threads on 1.9?",
"state":"closed",
"author":{
"id":1282691,
"login":"KristofferC"
},
"labels":[
{
"name":"linear algebra",
"description":"Linear algebra"
}
],
"created_date":"2022-12-31T18:49:47Z",
"updated_date":"2023-01-01T20:19:58Z",
"closed_date":"2023-01-01T20:19:57Z"
}
}
}

Activities: High-Level Intent Representation

To provide a more meaningful abstraction, actions are grouped into activities. Activities represent cohesive, high-level tasks performed by contributors, such as merging a pull request, publishing a release, or resolving an issue. This higher-level grouping removes noise from low-level event logs and aligns with the contributor's intent .

Activities are constructed based on logical and temporal criteria. For example, merging a pull request may involve several distinct actions: closing the pull request, pushing the merged changes, and deleting the source branch. By aggregating these actions, the activity more accurately reflects the contributor’s intent.

Each activity record represents a cohesive, high-level task and includes the following attributes:

activity: Specifies the type of activity (e.g., MergePullRequest, ReviewPullRequest, or PushCommits).

start_date: Indicates when the activity began, recorded in ISO 8601 format.

end_date: Indicates when the activity concluded, recorded in ISO 8601 format.

actor: Contains details about the contributor performing the activity, including a persistent id and their GitHub login.

repository: Provides details about the repository where the activity occurred, including its id, name, and associated organisation.

actions: A list of the actions that constitute the activity, retaining their original metadata for traceability.

The dataset includes 21 distinct activity types, which aggregate related actions based on logical and temporal criteria to represent contributors’ high-level intent:

AddContributors: Tracks the addition of one or more contributors to a repository within a short timeframe.

CloseIssue: Represents the resolution of an issue, optionally accompanied by a comment clarifying the closure.

ClosePullRequest: Indicates the closure of a pull request without merging its changes, optionally documented with a comment.

CommentCommits: Logs comments made directly on specific commits, often as part of discussions or reviews.

CommentIssue: Captures multiple comments on a specific issue.

CommentPullRequest: Records multiple inline or general comments on a pull request.

CreateRepository: Represents the creation of a new repository, optionally including the initialization of its main branch.

ForkRepository: Captures the action of creating a fork of an existing repository.

MakeRepositoryPublic: Tracks the transition of a private repository to public visibility.

ManageBranches: Logs the creation or deletion of branches within a repository.

ManageTags: Tracks the creation or deletion of tags, often linked to versioning or releases.

ManageWikiPages: Represents updates or edits to wiki pages associated with a repository.

MergePullRequest: Indicates the successful merging of a pull request, potentially accompanied by actions such as pushing changes, deleting branches, or closing linked issues.

OpenIssue: Logs the creation of a new issue within a repository to report bugs, request features, or raise concerns.

OpenPullRequest: Tracks the initiation of a pull request proposing changes to the repository.

PublishRelease: Represents the publication of a release, optionally involving the creation of a corresponding tag.

PushCommits: Logs a sequence of commits pushed to a branch within a repository.

ReopenIssue: Captures the reopening of a previously closed issue for further action, optionally accompanied by a clarifying comment.

ReopenPullRequest: Represents the reopening of a previously closed pull request for additional review or discussion, optionally with a comment.

ReviewPullRequest: Tracks the review process of a pull request, including general or inline comments and formal reviews.

StarRepository: Logs when a user stars a repository, signaling interest or support.

Example of activity record:

{
"activity":"MergePullRequest",
"start_date":"2023-01-01T20:19:57Z",
"end_date":"2023-01-01T20:20:05Z",
"actor":{
"id":1282691,
"login":"KristofferC"
},
"repository":{
"id":1644196,
"name":"JuliaLang/julia",
"organisation":"JuliaLang",
"organisation_id":743164
},
"actions":[
{
"action":"MergePullRequest",
"event_id":"26170139644",
"date":"2023-01-01T20:19:57Z",
"details":{
"pull_request":{
"id":1181521272,
"number":48064,
"title":"use the correct env variable name to set default openblas num threads",
"state":"closed",
"author":{
"id":1282691,
"login":"KristofferC"
},
"labels":[
{
"name":"backport 1.8",
"description":"Change should be backported to release-1.8"
},
{
"name":"backport 1.9",
"description":"Change should be backported to release-1.9"
}
],
"created_date":"2022-12-31T19:59:00Z",
"updated_date":"2023-01-01T20:19:57Z",
"closed_date":"2023-01-01T20:19:56Z",
"merged":true
}
}
},
{
"action":"CloseIssue",
"event_id":"26170139709",
"date":"2023-01-01T20:19:58Z",
"details":{
"issue":{
"id":1515182791,
"number":48062,
"title":"Bad default number of BLAS threads on 1.9?",
"state":"closed",
"author":{
"id":1282691,
"login":"KristofferC"
},
"labels":[
{
"name":"linear algebra",
"description":"Linear algebra"
}
],
"created_date":"2022-12-31T18:49:47Z",
"updated_date":"2023-01-01T20:19:58Z",
"closed_date":"2023-01-01T20:19:57Z"
}
}
},
{
"action":"DeleteBranch",
"event_id":"26170140410",
"date":"2023-01-01T20:20:04Z",
"details":{
"branch_name":"kc/openblas_threads"
}
},
{
"action":"PushCommits",
"event_id":"26170140428",
"date":"2023-01-01T20:20:05Z",
"details":{
"push":{
"id":12151296179,
"ref":"refs/heads/master",
"commits":1
}
}
}
]
}

NumFocus数据集全面呈现了NumFocus组织支持的58个开源项目的贡献者活动全貌。本数据集覆盖2022年1月至2024年12月共三年的观测周期，完整记录了由科学与数据驱动型软件项目组成的特定社区内的开源协作动态。

为解决原始GitHub事件日志的解读难题，本数据集设计了两级结构化抽象层级：操作（Actions）与活动（Activities）。其中操作级详细记录单一步骤行为，例如创建分支、推送提交；活动级则将相关操作聚合为高阶任务，例如合并拉取请求（pull request）、解决议题（issue）。该层级架构填补了细粒度操作与贡献者整体意图之间的鸿沟。

本数据集主体以活动为核心，提供贡献者行为的高阶概览；针对需要细粒度分析的用户，还附带了操作级的补充数据集。

本数据集配套一款基于Python的命令行工具，可自动将原始GitHub事件日志转换为结构化的操作与活动数据。该工具及其可配置映射文件、脚本均已通过ghmap公开获取。

本数据集由以下文件组成：
- NumFocus_Jan22-Dec24_GH_Actions.zip：包含2,716,910条JSON Lines格式的操作数据，记录单条贡献者操作行为。
- NumFocus_Jan22-Dec24_GH_Activities.zip：包含2,278,299条JSON Lines格式的活动数据，代表由聚合操作生成的高阶任务。
- action_schema.json：JSON格式的验证模式，用于确保操作数据集解读的一致性。
- activity_schema.json：JSON格式的验证模式，用于活动数据集的验证与集成。

### 操作（Actions）：细粒度底层操作
操作是记录的最细粒度操作层级，源自原始GitHub事件。每条操作对应一项明确的贡献者操作，例如向仓库（repository）推送代码、发起拉取请求（pull request）或对议题（issue）发表评论。该层级保留了追踪单条操作所需的全部技术细节，同时对事件数据进行标准化处理，以支持跨仓库的分析工作。

每条操作记录对应一项贡献者操作，包含以下属性：
- `action`：指定操作类型（例如PushCommits、OpenPullRequest或CreateBranch）。
- `event_id`：将操作与其原始GitHub事件关联的唯一标识符。
- `date`：操作的时间戳，采用ISO 8601格式记录。
- `actor`：包含执行该操作的贡献者详情，包括持久化ID与GitHub用户名。
- `repository`：记录操作发生所在仓库的信息，包括仓库ID、名称与所属组织（organisation）。
- `details`：存储与操作类型相关的额外属性，源自对应GitHub事件的负载数据（例如，PushCommits操作的`details`包含分支引用与提交数量；OpenPullRequest操作的`details`包含拉取请求的标题、标签、状态以及创建与更新时间）。

本数据集涵盖24种不同的操作类型，每种均源自特定GitHub事件，对应一项明确的贡献者操作：
- AddMember：记录向仓库新增协作者的操作。
- CloseIssue：标记议题已由贡献者关闭。
- ClosePullRequest：代表未合并变更的拉取请求被关闭。
- CommentCommit：捕获针对仓库内特定提交的评论。
- CreateBranch：记录仓库内新分支的创建。
- CreateIssueComment：追踪针对现有议题的评论。
- CreatePullRequestComment：记录针对拉取请求的评论，包括对所提变更的讨论。
- CreatePullRequestReview：代表提交拉取请求评审。
- CreatePullRequestReviewComment：捕获拉取请求评审过程中添加的内联评论。
- CreateRepository：代表新GitHub仓库的创建。
- CreateTag：记录标签的创建，通常与版本控制或发布相关。
- DeleteBranch：指示仓库内现有分支已被删除。
- DeleteTag：追踪仓库内标签的删除操作。
- ForkRepository：捕获将仓库复刻至其他账户下的操作。
- MakeRepositoryPublic：将私有仓库的可见性变更为公开。
- ManageWikiPage：记录仓库Wiki页面的编辑或更新。
- MergePullRequest：指示拉取请求已被合并，将其变更整合至基础分支。
- OpenIssue：捕获仓库内新议题的创建。
- OpenPullRequest：代表发起新拉取请求以提议变更。
- PublishRelease：追踪发布版本的发布操作，通常与特定标签及关联元数据绑定。
- PushCommits：记录推送事件，详细说明操作涉及的分支与提交。
- ReopenIssue：指示此前关闭的议题已被重新打开以开展后续操作。
- ReopenPullRequest：捕获此前关闭的拉取请求已被重新打开。
- StarRepository：记录用户为仓库加星标以收藏或表达支持的行为。

#### 操作记录示例：
json
{
"action":"CloseIssue",
"event_id":"26170139709",
"date":"2023-01-01T20:19:58Z",
"actor":{
"id":1282691,
"login":"KristofferC"
},
"repository":{
"id":1644196,
"name":"JuliaLang/julia",
"organisation":"JuliaLang",
"organisation_id":743164
},
"details":{
"issue":{
"id":1515182791,
"number":48062,
"title":"Bad default number of BLAS threads on 1.9?",
"state":"closed",
"author":{
"id":1282691,
"login":"KristofferC"
},
"labels":[
{
"name":"linear algebra",
"description":"Linear algebra"
}
],
"created_date":"2022-12-31T18:49:47Z",
"updated_date":"2023-01-01T20:19:58Z",
"closed_date":"2023-01-01T20:19:57Z"
}
}
}

### 活动（Activities）：高阶意图表征
为提供更具意义的抽象层级，操作被聚合为活动。活动代表贡献者执行的连贯高阶任务，例如合并拉取请求、发布版本或解决议题。该高阶聚合操作可过滤底层事件日志中的冗余信息，并与贡献者的真实意图保持一致。

活动基于逻辑与时间标准构建。例如，合并拉取请求可能包含多个独立操作：关闭拉取请求、推送合并后的变更、删除源分支。通过聚合这些操作，活动可更精准地反映贡献者的意图。

每条活动记录代表一项连贯的高阶任务，包含以下属性：
- `activity`：指定活动类型（例如MergePullRequest、ReviewPullRequest或PushCommits）。
- `start_date`：活动开始时间，采用ISO 8601格式记录。
- `end_date`：活动结束时间，采用ISO 8601格式记录。
- `actor`：包含执行该活动的贡献者详情，包括持久化ID与GitHub用户名。
- `repository`：记录活动发生所在仓库的信息，包括仓库ID、名称与所属组织。
- `actions`：构成该活动的操作列表，保留原始元数据以支持溯源。

本数据集涵盖21种不同的活动类型，这些类型基于逻辑与时间标准聚合相关操作，以表征贡献者的高阶意图：
- AddContributors：记录短时间内向仓库新增一名或多名贡献者的操作。
- CloseIssue：代表议题已解决，可附带用于说明关闭原因的评论。
- ClosePullRequest：指示未合并变更的拉取请求已关闭，可附带相关评论。
- CommentCommits：记录针对特定提交的评论，通常属于讨论或评审环节。
- CommentIssue：捕获针对单个议题的多条评论。
- CommentPullRequest：记录针对拉取请求的多条内联或通用评论。
- CreateRepository：代表新仓库的创建，可包含初始化主分支的操作。
- ForkRepository：捕获将现有仓库复刻至其他账户的操作。
- MakeRepositoryPublic：追踪私有仓库转为公开可见的过程。
- ManageBranches：记录仓库内分支的创建或删除。
- ManageTags：追踪标签的创建或删除，通常与版本控制或发布相关。
- ManageWikiPages：代表仓库关联Wiki页面的更新或编辑。
- MergePullRequest：指示拉取请求已成功合并，可伴随推送变更、删除分支或关闭关联议题等操作。
- OpenIssue：记录仓库内新议题的创建，用于报告bug、请求功能或提出关切。
- OpenPullRequest：追踪发起拉取请求以提议仓库变更的操作。
- PublishRelease：代表发布版本的发布，可包含对应标签的创建。
- PushCommits：记录向仓库内某一分支推送的一系列提交。
- ReopenIssue：捕获此前关闭的议题已被重新打开以开展后续操作，可附带说明性评论。
- ReopenPullRequest：代表此前关闭的拉取请求已被重新打开以接受额外评审或讨论，可附带相关评论。
- ReviewPullRequest：追踪拉取请求的评审过程，包括通用或内联评论以及正式评审。
- StarRepository：记录用户为仓库加星标的行为，以表达兴趣或支持。

#### 活动记录示例：
json
{
"activity":"MergePullRequest",
"start_date":"2023-01-01T20:19:57Z",
"end_date":"2023-01-01T20:20:05Z",
"actor":{
"id":1282691,
"login":"KristofferC"
},
"repository":{
"id":1644196,
"name":"JuliaLang/julia",
"organisation":"JuliaLang",
"organisation_id":743164
},
"actions":[
{
"action":"MergePullRequest",
"event_id":"26170139644",
"date":"2023-01-01T20:19:57Z",
"details":{
"pull_request":{
"id":1181521272,
"number":48064,
"title":"use the correct env variable name to set default openblas num threads",
"state":"closed",
"author":{
"id":1282691,
"login":"KristofferC"
},
"labels":[
{
"name":"backport 1.8",
"description":"Change should be backported to release-1.8"
},
{
"name":"backport 1.9",
"description":"Change should be backported to release-1.9"
}
],
"created_date":"2022-12-31T19:59:00Z",
"updated_date":"2023-01-01T20:19:57Z",
"closed_date":"2023-01-01T20:19:56Z",
"merged":true
}
}
},
{
"action":"CloseIssue",
"event_id":"26170139709",
"date":"2023-01-01T20:19:58Z",
"details":{
"issue":{
"id":1515182791,
"number":48062,
"title":"Bad default number of BLAS threads on 1.9?",
"state":"closed",
"author":{
"id":1282691,
"login":"KristofferC"
},
"labels":[
{
"name":"linear algebra",
"description":"Linear algebra"
}
],
"created_date":"2022-12-31T18:49:47Z",
"updated_date":"2023-01-01T20:19:58Z",
"closed_date":"2023-01-01T20:19:57Z"
}
}
},
{
"action":"DeleteBranch",
"event_id":"26170140410",
"date":"2023-01-01T20:20:04Z",
"details":{
"branch_name":"kc/openblas_threads"
}
},
{
"action":"PushCommits",
"event_id":"26170140428",
"date":"2023-01-01T20:20:05Z",
"details":{
"push":{
"id":12151296179,
"ref":"refs/heads/master",
"commits":1
}
}
}
]
}