seongchaeae/sage-pretrain-corpus

Sage Pretrain Corpus (v0.8)是一个用于韩语LLM项目“Sage”的预训练语料库，经过许可证审核，包含韩语和英语数据。该数据集总计128,305,142个文档，每个文档包含以下字段：id、text、source、license和meta（原始字段的JSON）。数据来源多样，包括FineWeb-2 Korean、Cosmopedia、OpenMathInstruct-2、FineWeb-Edu、FineMath、OpenR1-Math、Project Gutenberg、NuminaMath-1.5、Bespoke-Stratos-17k、GSM8K、竞争数学数据集、CodeParrot清洁代码、韩语和英语维基百科、peS2o CS、arXiv元数据、ACL Anthology和Common Pile等。数据集采用混合许可证（标记为“other”），用户需遵守每个来源的单独许可证，例如CC-BY要求署名，CC-BY-SA要求相同方式共享。v0.8版本新增了三种推理数据：Bespoke-Stratos-17k（DeepSeek-R1蒸馏推理痕迹）、GSM8K（小学数学习题分步解答）和竞争数学数据集（数学竞赛问题）。数据集旨在为韩语LLM提供高质量、多样化的预训练数据，覆盖教育、数学、代码、科学、文学等多个领域，并经过严格筛选以确保质量和许可证合规性。

Sage Pretrain Corpus (v0.8) is a license-audited Korean/English pretraining corpus for the Sage Korean LLM project. It contains a total of 128,305,142 documents, each with fields: id, text, source, license, and meta (JSON of original fields). The dataset is sourced from multiple origins, including FineWeb-2 Korean, Cosmopedia, OpenMathInstruct-2, FineWeb-Edu, FineMath, OpenR1-Math, Project Gutenberg, NuminaMath-1.5, Bespoke-Stratos-17k, GSM8K, competition math datasets, CodeParrot clean code, Korean and English Wikipedia, peS2o CS, arXiv metadata, ACL Anthology, and Common Pile. It uses mixed licenses (labeled as other), and users must comply with each sources individual license, e.g., CC-BY requires attribution, and CC-BY-SA carries ShareAlike. Version v0.8 adds three reasoning data sources: Bespoke-Stratos-17k (DeepSeek-R1 distilled reasoning traces), GSM8K (grade school math step-by-step solutions), and competition math datasets (math competition problems). The corpus aims to provide high-quality, diverse pretraining data for Korean LLMs, covering domains such as education, mathematics, code, science, and literature, with rigorous filtering for quality and license compliance.