paper-sub/neurips-3231

Impermanent是一个实时基准测试，用于评估时间序列预测模型在开放世界时间变化下的表现。与具有固定训练-测试分割的静态基准不同，它通过连续更新的数据流按时间顺序对预测进行评分，从而能够研究时间鲁棒性、分布偏移和性能稳定性，而不是在固定测试集上的一次性准确性。该基准基于GitHub开源活动数据（来自GH Archive），覆盖了按星标数排名的前400个仓库，跟踪三种预测频率（每日、每周、每月）下的四种事件类型（星标、打开的问题、打开的拉取请求、推送事件），并具有每日更新和标准化排行榜，以支持可重复的持续比较。数据集包含三个配置，分别对应不同的预测频率，每个配置由Parquet文件组成，数据从2023年1月1日开始，并持续更新，首个发布版本中的评估窗口涵盖从2025年10月1日到2026年4月19日的20个截止点。Impermanent is a live benchmark that evaluates time-series forecasting models under open-world temporal change. Unlike static benchmarks with fixed train-test splits, Impermanent scores forecasts sequentially over time on continuously updated data streams, enabling the study of temporal robustness, distributional shift, and performance stability rather than one-off accuracy on a frozen test set. The benchmark is instantiated on GitHub open-source activity data sourced from GH Archive, covering the top 400 repositories by star count and tracking four event types across three forecast frequencies (daily, weekly, monthly), with daily updates and standardized leaderboards for reproducible, ongoing comparison. The dataset includes three configurations corresponding to the forecast frequencies, each consisting of Parquet files, with data starting from 2023-01-01 and continuously updated, and the evaluation window in the first release spans 20 cutoffs from 2025-10-01 to 2026-04-19.