Context is Key (CiK)

Context is Key (CiK) 是一个时间序列预测基准数据集，由ServiceNow Research等机构创建。该数据集包含71个预测任务，涵盖7个应用领域，结合了数值数据和精心设计的文本上下文，要求模型整合这两种模态。数据集的内容包括来自公共来源的2,644个真实世界的时间序列数据和部分模拟数据，采样频率从每10分钟到每月不等。数据集的创建过程涉及手动筛选和调整数据，确保上下文信息的质量和相关性。CiK数据集主要用于评估和提升多模态预测模型的性能，旨在解决在预测中有效整合文本上下文信息的问题。Context is Key (CiK) is a time series forecasting benchmark dataset developed by institutions including ServiceNow Research. It includes 71 forecasting tasks across 7 application domains, combining numerical data and carefully curated textual contexts, and requires models to integrate these two modalities. The dataset contains 2,644 real-world time series datasets from public sources plus some simulated data, with sampling frequencies ranging from every 10 minutes to monthly. The creation process of CiK involves manual screening and data refinement to ensure the quality and relevance of contextual information. Primarily, the CiK dataset is used to evaluate and enhance the performance of multimodal forecasting models, with the goal of addressing the challenge of effectively integrating textual contextual information during forecasting.

搜集汇总

数据集介绍

构建方式

Context is Key (CiK) 数据集通过精心设计，将数值数据与多样化的文本上下文配对，构建了一个时间序列预测基准。该数据集包含71个预测任务，涵盖7个领域，每个任务都要求模型整合数值数据和文本上下文。数据集的构建过程包括手动筛选和发布任务，确保每个任务都包含不同类型的上下文信息，并需要多种能力来充分利用上下文信息。此外，数据集还引入了一个新的评估指标——Region of Interest CRPS (RCRPS)，该指标优先考虑上下文敏感的时间窗口，并考虑约束满足情况。

特点

CiK 数据集的特点在于其多模态性，结合了数值数据和文本上下文，要求模型在预测时必须整合这两种信息。数据集涵盖了多个领域，包括气候学、经济学、能源、机械、公共安全、交通和零售，每个领域都有其特定的上下文信息。此外，数据集设计了多种类型的上下文信息，如过去事件、已知因果关系等，并要求模型具备多种能力，如因果推理、数学推理等，以解锁准确的预测。

使用方法

CiK 数据集的使用方法包括评估各种预测方法，包括统计模型、时间序列基础模型和基于大型语言模型（LLM）的预测模型。数据集提供了一个可视化平台，用户可以在 https://servicenow.github.io/context-is-key-forecasting/v0/ 上探索任务和源代码。使用该数据集时，研究人员可以比较不同模型在整合上下文信息方面的表现，并探索新的预测方法，特别是那些能够有效利用文本上下文的模型。

背景与挑战

背景概述

The Context is Key (CiK) benchmark was introduced to address the critical need for accurate forecasting by integrating essential textual information with numerical data. Developed by researchers from ServiceNow Research, Mila - Québec AI Institute, Université de Montréal, Polytechnique Montréal, Université Laval, University of Toronto, and McGill University, the CiK benchmark aims to evaluate the ability of forecasting models to effectively utilize both numerical data and carefully crafted textual context. The core research question revolves around how existing forecasting models can effectively integrate textual information, which remains an open question. The CiK benchmark pairs numerical data with diverse types of textual context, requiring models to integrate both modalities. This approach aims to advance multimodal forecasting, promoting models that are both accurate and accessible to decision-makers with varied technical expertise.

当前挑战

The CiK benchmark presents several challenges. Firstly, it addresses the challenge of integrating textual context with numerical data to improve forecasting accuracy. This involves the challenge of ensuring that models can effectively interpret and utilize contextual information provided in natural language. Secondly, the benchmark highlights the challenge of constructing a dataset that pairs numerical data with relevant textual context, ensuring that the context is not only descriptive but also useful for improving forecasting performance. Additionally, the benchmark faces the challenge of evaluating models that can jointly leverage historical observations and natural language for forecasting, as there are currently no systematic evaluations of these models' abilities. Finally, the benchmark must mitigate the risk of memorization by pre-trained models, which could potentially inflate evaluation performance, by employing strategies such as using live data sources and applying minor transformations to the data.

常用场景

经典使用场景

Context is Key (CiK) 数据集的经典使用场景在于评估时间序列预测模型在整合关键文本信息方面的能力。该数据集通过将数值数据与精心构建的文本上下文配对，要求模型同时利用这两种模态进行预测。经典的使用场景包括评估统计模型、时间序列基础模型以及基于大型语言模型（LLM）的预测模型，并提出了一种简单而有效的LLM提示方法，该方法在基准测试中优于所有其他测试方法。

实际应用

CiK 数据集的实际应用场景广泛，涵盖了从气候学、经济学到能源和零售等多个领域。例如，在气候学中，模型可以利用文本描述的天气条件来改进太阳能发电量的预测；在经济学中，模型可以结合政策变化或市场新闻来提高失业率预测的准确性。此外，CiK 数据集还可以用于训练和评估能够自动选择模型并整合先验信息的机器学习模型，从而实现时间序列预测的自动化和普及化。

衍生相关工作

CiK 数据集的引入催生了一系列相关工作，特别是在多模态时间序列预测领域。例如，一些研究工作探索了如何将大型语言模型（LLM）重新用于预测，并利用自然语言作为整合侧面信息的直观接口。此外，还有研究致力于开发能够处理时间序列数据的预训练LLM，通过引入特殊标记或修改编码器来适应时间序列数据。这些工作不仅展示了CiK 数据集在推动创新方面的潜力，还揭示了现有方法在整合文本信息方面的局限性。