LeadLagForecasting/llf_github

---
dataset_info:
features:
- name: id
dtype: string
- name: platforms
sequence: string
- name: created_date
dtype: timestamp[ms]
- name: first_create
dtype: float64
- name: first_push
dtype: float64
- name: first_star
dtype: float64
- name: first_fork
dtype: float64
- name: cumulative_pushes_w_creates
sequence: int64
- name: cumulative_pushes
sequence: int64
- name: cumulative_stars
sequence: int64
- name: cumulative_forks
sequence: int64
splits:
- name: train
num_bytes: 87650021696
num_examples: 938567
- name: labeled_test
num_bytes: 21930381181
num_examples: 234640
- name: val
num_bytes: 21906489783
num_examples: 234615
- name: unlabeled_test
num_bytes: 50784636048
num_examples: 1568622
- name: train_extra
num_bytes: 6294971369
num_examples: 58077
download_size: 1094625852
dataset_size: 188566500077
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
- split: labeled_test
path: data/labeled_test-*
- split: val
path: data/val-*
- split: unlabeled_test
path: data/unlabeled_test-*
- split: train_extra
path: data/train_extra-*
viewer_feature_names_to_ignore_statistics:
- cumulative_pushes_w_creates
- cumulative_pushes
- cumulative_stars
- cumulative_forks
license: cc-by-4.0
task_categories:
- time-series-forecasting
---

# Dataset Card for Lead-Lag Forecasting Benchmark: GitHub Dataset

This dataset is part of the benchmark introduced by the paper **[Benchmark Datasets for Lead-Lag Forecasting on Social Platforms](https://arxiv.org/abs/2511.03877)**. It provides time series data for over 3 million GitHub repositories, for modeling and predicting long-term impact (quantified by cumulative forks) from short-term signals (pushes and stars). Please visit our project page https://lead-lag-forecasting.github.io for more information.

## Paper Abstract

Social and collaborative platforms emit multivariate time-series traces in which early interactions-such as views, likes, or downloads-are followed, sometimes months or years later, by higher impact like citations, sales, or reviews. We formalize this setting as Lead-Lag Forecasting (LLF): given an early usage channel (the lead), predict a correlated but temporally shifted outcome channel (the lag). Despite the ubiquity of such patterns, LLF has not been treated as a unified forecasting problem within the time-series community, largely due to the absence of standardized datasets. To anchor research in LLF, here we present two high-volume benchmark datasets-arXiv (accesses → citations of 2.3M papers) and GitHub (pushes/stars → forks of 3M repositories)-and outline additional domains with analogous lead-lag dynamics, including Wikipedia (page views → edits), Spotify (streams → concert attendance), e-commerce (click-throughs → purchases), and LinkedIn profile (views → messages). Our datasets provide ideal testbeds for lead-lag forecasting, by capturing long-horizon dynamics across years, spanning the full spectrum of outcomes, and avoiding survivorship bias in sampling. We documented all technical details of data curation and cleaning, verified the presence of lead-lag dynamics through statistical and classification tests, and benchmarked parametric and non-parametric baselines for regression. Our study establishes LLF as a novel forecasting paradigm and lays an empirical foundation for its systematic exploration in social and usage data.

## Dataset Features

| Name | Type | Description | Extraction / Notes |
| :--- | :--- | :--- | :--- |
| **id** | `string` | Repository identifier: `owner_name/repo_name` | From Ecosyste.ms metadata |
| **platforms** | `list[string]` | List of platforms the repo was released on (e.g., PyPI, npm) | From Ecosyste.ms metadata |
| **created_date** | `timestamp` | Date of GitHub repository creation | From Ecosyste.ms metadata |
| **first_create** | `float` | Days between first `CreateEvent` and creation date. `!=0` implies anomaly. | Comparison between GH Archive and Ecosyste.ms |
| **first_push** | `float` | Days between first `PushEvent` and creation date. `<0` implies anomaly. | Comparison between GH Archive and Ecosyste.ms |
| **first_star** | `float` | Days between first `WatchEvent` (star) and creation date. `<0` implies anomaly. | Comparison between GH Archive and Ecosyste.ms |
| **first_fork** | `float` | Days between first `ForkEvent` and creation date. `<0` implies anomaly. | Comparison between GH Archive and Ecosyste.ms |
| **cumulative_pushes_w_creates** | `list[int]` | Daily cumulative pushes + creates (of repo, branches, and tags) since creation date. | Summed per day from GH Archive events |
| **cumulative_pushes** | `list[int]` | Daily cumulative pushes without `CreateEvents` since creation date. | Summed per day from GH Archive events |
| **cumulative_stars** | `list[int]` | Daily cumulative `WatchEvents` (stars) since creation date. | Summed per day from GH Archive events |
| **cumulative_forks** | `list[int]` | Daily cumulative forks since creation date. | Summed per day from GH Archive events |

## Dataset Splits and Statistics

| Split Name | Description | Number of Examples |
| :--- | :--- | :--- |
| **train** | Random sample from labeled packages with >= years of activity and no anomalies. | 938,567 |
| **val** | Random validation subset. | 234,615 |
| **labeled_test** | Random test subset. | 234,640 |
| **unlabeled_test** | Repositories with < 5 years of activity. | 1,568,622 |
| **train_extra** | Repositories with anomalies. | 58,077 |

## Citation

If you use this dataset in your research, please cite the following paper:

```bibtex
@article{kazemian2025benchmark,
title={Benchmark Datasets for Lead-Lag Forecasting on Social Platforms},
author={Kazemian, Kimia and Liu, Zhenzhen and Yang, Yangfanyu and Luo, Katie Z and Gu, Shuhan and Du, Audrey and Yang, Xinyu and Jansons, Jack and Weinberger, Kilian Q and Thickstun, John and others},
journal={arXiv preprint arXiv:2511.03877},
year={2025}
}
```

dataset_info:
数据集信息:
特征:
- 名称: id
数据类型: 字符串
- 名称: platforms
数据类型: 字符串序列
- 名称: created_date
数据类型: 毫秒级时间戳
- 名称: first_create
数据类型: float64
- 名称: first_push
数据类型: float64
- 名称: first_star
数据类型: float64
- 名称: first_fork
数据类型: float64
- 名称: cumulative_pushes_w_creates
数据类型: int64序列
- 名称: cumulative_pushes
数据类型: int64序列
- 名称: cumulative_stars
数据类型: int64序列
- 名称: cumulative_forks
数据类型: int64序列
数据集划分:
- 名称: train
字节数: 87650021696
样本量: 938567
- 名称: labeled_test
字节数: 21930381181
样本量: 234640
- 名称: val
字节数: 21906489783
样本量: 234615
- 名称: unlabeled_test
字节数: 50784636048
样本量: 1568622
- 名称: train_extra
字节数: 6294971369
样本量: 58077
下载总大小: 1094625852 字节
数据集总大小: 188566500077 字节
配置:
- 配置名称: default
数据文件:
- 划分: train
路径: data/train-*
- 划分: labeled_test
路径: data/labeled_test-*
- 划分: val
路径: data/val-*
- 划分: unlabeled_test
路径: data/unlabeled_test-*
- 划分: train_extra
路径: data/train_extra-*
可视化需忽略统计信息的特征:
- cumulative_pushes_w_creates
- cumulative_pushes
- cumulative_stars
- cumulative_forks
许可证: cc-by-4.0
任务类别:
- 时间序列预测（time-series-forecasting）

# 领先滞后预测基准数据集：GitHub数据集
本数据集属于论文**《面向社交平台的领先滞后预测基准数据集》（[Benchmark Datasets for Lead-Lag Forecasting on Social Platforms](https://arxiv.org/abs/2511.03877)）**所提出的基准数据集之一。该数据集提供了超过300万个GitHub仓库的时间序列数据，用于基于短期信号（推送与标星行为）建模并预测长期影响力（以累计复刻量量化）。如需更多信息，请访问项目页面https://lead-lag-forecasting.github.io。

## 论文摘要
社交与协作平台会产生多变量时间序列轨迹：早期交互行为（如浏览、点赞或下载）之后，往往会在数月甚至数年后出现更高影响力的结果，例如引用量、销售额或评论数。我们将这一场景形式化为**领先滞后预测（Lead-Lag Forecasting, LLF）**：给定早期使用渠道（领先变量），预测存在相关性但存在时间偏移的结果渠道（滞后变量）。尽管此类模式广泛存在，但领先滞后预测尚未被时间序列社区视为统一的预测问题，主要原因在于缺乏标准化的基准数据集。为锚定领先滞后预测领域的研究，本文提出两个大规模基准数据集——ArXiv数据集（访问量→230万篇论文的引用量）与GitHub数据集（推送/标星→300万个仓库的复刻量），同时梳理了其他具备类似领先滞后动态的领域，包括维基百科（页面浏览量→编辑量）、Spotify（流媒体播放量→演唱会到场人数）、电子商务（点击率→购买量）以及LinkedIn主页（浏览量→私信数）。本数据集通过捕捉跨多年的长时序动态、覆盖全范围的结果变量，并在采样过程中避免了生存偏差，为领先滞后预测任务提供了理想的测试平台。我们详细记录了数据整理与清洗的全部技术细节，通过统计与分类测试验证了领先滞后动态的存在性，并为回归任务基准测试了参数化与非参数化基线模型。本研究将领先滞后预测确立为一种全新的预测范式，并为社交与使用数据中的系统探索奠定了实证基础。

## 数据集特征
| 名称 | 类型 | 描述 | 提取/说明 |
| :--- | :--- | :--- | :--- |
| **id** | `string` | 仓库标识符：格式为`所有者名称/仓库名称` | 来自Ecosyste.ms元数据 |
| **platforms** | `list[string]` | 仓库发布所在的平台列表（例如PyPI、npm） | 来自Ecosyste.ms元数据 |
| **created_date** | `timestamp` | GitHub仓库创建日期 | 来自Ecosyste.ms元数据 |
| **first_create** | `float` | 首次`CreateEvent`与仓库创建日期之间的天数，若值不为0则说明存在异常 | 对比GH Archive与Ecosyste.ms数据所得 |
| **first_push** | `float` | 首次`PushEvent`与仓库创建日期之间的天数，若值小于0则说明存在异常 | 对比GH Archive与Ecosyste.ms数据所得 |
| **first_star** | `float` | 首次`WatchEvent`（标星）与仓库创建日期之间的天数，若值小于0则说明存在异常 | 对比GH Archive与Ecosyste.ms数据所得 |
| **first_fork** | `float` | 首次`ForkEvent`与仓库创建日期之间的天数，若值小于0则说明存在异常 | 对比GH Archive与Ecosyste.ms数据所得 |
| **cumulative_pushes_w_creates** | `list[int]` | 自仓库创建之日起，每日累计推送（含创建事件，涵盖仓库、分支与标签的创建）总量 | 来自GH Archive事件的每日汇总 |
| **cumulative_pushes** | `list[int]` | 自仓库创建之日起，每日累计推送（不含CreateEvents）总量 | 来自GH Archive事件的每日汇总 |
| **cumulative_stars** | `list[int]` | 自仓库创建之日起，每日累计WatchEvent（标星）总量 | 来自GH Archive事件的每日汇总 |
| **cumulative_forks** | `list[int]` | 自仓库创建之日起，每日累计复刻（Fork）总量 | 来自GH Archive事件的每日汇总 |

## 数据集划分与统计信息
| 划分名称 | 描述 | 样本数量 |
| :--- | :--- | :--- |
| **train** | 从具备至少一年活跃记录且无异常的带标签仓库中随机采样得到 | 938,567 |
| **val** | 随机验证子集 | 234,615 |
| **labeled_test** | 随机测试子集 | 234,640 |
| **unlabeled_test** | 活跃时长不足5年的仓库 | 1,568,622 |
| **train_extra** | 存在异常的仓库 | 58,077 |

## 引用
若在研究中使用本数据集，请引用以下论文：
bibtex
@article{kazemian2025benchmark,
title={Benchmark Datasets for Lead-Lag Forecasting on Social Platforms},
author={Kazemian, Kimia and Liu, Zhenzhen and Yang, Yangfanyu and Luo, Katie Z and Gu, Shuhan and Du, Audrey and Yang, Xinyu and Jansons, Jack and Weinberger, Kilian Q and Thickstun, John and others},
journal={arXiv preprint arXiv:2511.03877},
year={2025}
}