utter-project/LongBlocks

LongBlocks 是一个用于训练长上下文语言模型的多语言合成数据集。它包含大约194K个长上下文问答示例，这些示例生成长文档，涵盖书籍、网络文本、维基百科、arXiv论文、编程/代码数据以及社区问答。该数据集旨在支持需要基于扩展输入进行推理的任务的长上下文适应，包括：长文档的多跳推理、上下文基础、长形式问答、信息合成、文档级理解以及长上下文下的代码和技术推理。每个示例包括一个长源文档、一个合成问题、一个参考答案以及来自几个教师模型的可选模型生成响应。数据集设计用于通过监督微调、蒸馏或强化学习对长上下文LLM进行后训练。数据来源包括Institutional-Books-1.0、FineWeb2-HQ、PGBooks、ArXiv、Wikipedia、Stack-Edu和StackExchange。数据集结构包含一个训练分割，约194K行，列包括id、document、source、language、question、answer以及多个模型响应列。数据集在CC BY-SA 4.0许可下发布，但某些源语料库可能有额外限制。LongBlocks is a multilingual synthetic dataset developed for training long-context language models. It comprises approximately 194K long-context question-answering (QA) examples, which are constructed using long-form documents spanning books, web text, Wikipedia, arXiv papers, programming/code data, and community QA content. This dataset is aimed at supporting long-context adaptation for tasks that demand reasoning over extended inputs, including multi-hop reasoning across long documents, context grounding, long-form QA, information synthesis, document-level comprehension, as well as code and technical reasoning within long-context settings. Each example consists of a long source document, a synthetic question, a reference answer, and optional model-generated responses from multiple teacher models. The dataset is designed for post-training of long-context LLMs through supervised fine-tuning, knowledge distillation, or reinforcement learning. The data sources include Institutional-Books-1.0, FineWeb2-HQ, PGBooks, ArXiv, Wikipedia, Stack-Edu, and StackExchange. The dataset structure features a single training split with roughly 194K rows, with columns including id, document, source, language, question, answer, and multiple model response columns. The dataset is released under the CC BY-SA 4.0 license, although certain source corpora may have additional restrictions.