KairosQA

# KairosQA Dataset <p align="center"> <img src="kairosQA_viz.png" width="500" alt="KairosQA Logo"> </p> ## Dataset Description **KairosQA** is a temporally grounded question-answering dataset designed to evaluate the temporal alignment and reasoning capabilities of Large Language Models (LLMs). Unlike static benchmarks, KairosQA focuses on facts that evolve over time, specifically subject–relation–object triplets from Wikidata that changed at least twice between 2018 and 2025 as described in our paper [Understanding Data Temporality Impact on Large Language Models Pre-training](https://arxiv.org/abs/2605.22769) and our [blogpost](https://kyutai.org/blog/2026-05-26-kairos). The dataset prioritizes "popular" subjects (based on Wikipedia page views) to ensure that evaluation failures are a result of temporal misalignment rather than a simple lack of world knowledge. It was first designed to assess the improvements in terms of freshness of temporally ordered data pretraining compared to shuffled pretraining data. Yearly checkpoints from sequential training are available here: [Sequential Helium 6B](https://huggingface.co/kyutai/Sequential_Helium_6B). --- ## Dataset Summary * **Source:** Wikidata (filtered for temporal variation). * **Total Subject-Relation Pairs:** 7,167. * **Focus Areas:** Sports, organizations, career-related facts, and events. * **Format:** Multiple-choice questions (MCQs) generated via GPT-4o mini. * **Temporal Range:** Evaluation snapshots from 2014 to 2025. --- ## Dataset Construction ### 1. Filtering & Popularity From an initial pool of 17 million triplets, successive filters were applied to ensure temporal validity. The dataset focuses on the top 20% most popular subjects to maintain a high information density. **Quantile Distribution of Subject Popularity** | Datasets | MIN | 25% | 50% | 75% | MAX | | :--- | :--- | :--- | :--- | :--- | :--- | | **Wikidata** | 1 | 340 | 1,378 | 6,151 | 133,423,662 | | **KairosQA** | 161 | 27,277 | 78,194 | 320,176 | 44,800,923 | ### 2. Temporal Distribution The number of available evaluation examples varies by year, ensuring that for any given "target year," the model is only tested against facts with a confirmed ground-truth answer for that specific timestamp. | Years | 2014 | 2017 | 2020 | 2022 | 2024 | 2025 | | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | **Nb of Examples** | 4,718 | 5,844 | 6,628 | 6,762 | 6,315 | 6,073 | ### 3. Generation Multiple-choice questions were created using a few-shot prompting approach. * **Question Variety:** LLMs modified reference templates to incorporate the target year naturally. * **Distractors:** Hard distractors were drawn from "neighboring years" (e.g., an athlete's former team) to challenge the model's ability to distinguish chronological nuances. --- ## Citations If you use this dataset, please cite: ```bibtex @misc{pilchen2026understandingdatatemporalityimpact, title={Understanding Data Temporality Impact on Large Language Models Pre-training}, author={Hippolyte Pilchen and Romain Fabre and Franck Signe Talla and Patrick Perez and Edouard Grave}, year={2026}, eprint={2605.22769}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2605.22769}, } ```