alea-institute/kl3m-data-eu-oj

KL3M数据集是ALEA研究所提供的用于大型语言模型训练的版权清理资源，包含超过1.32亿个文档和数万亿个标记，涵盖了16个不同的来源。该数据集完全遵守版权法和合同条款，并以公共领域材料、美国政府作品、开放获取内容以及明确授权用于AI训练的内容为基础。数据集采用Parquet文件格式，包含了文档文本和元数据信息。

The KL3M dataset is a set of copyright-clean training resources for large language models provided by the ALEA Institute, containing over 132 million documents and trillions of tokens from 16 different sources. The dataset is fully compliant with copyright law and contractual terms, based on public domain materials, US government works, open access content, and content explicitly licensed for AI training. The dataset is formatted in Parquet files, including document text and metadata.