Zyphra/Zyda-2

Zyda-2是一个包含5万亿个标记的语言建模数据集，由Zyphra公司策划，主要用于文本生成任务。该数据集通过收集和交叉去重多个高质量开源数据集（包括Zyda、FineWeb、DCLM和Dolma）构建而成，涵盖了网络数据、教育内容、数学、代码和科学论文等多种来源。Zyda-2在模型训练中表现出色，尤其在教育和自然语言推理内容方面。数据集的主要语言为英语，发布在Open Data Commons License许可下。

Zyda-2 is a 5 trillion token language modeling dataset created by combining high-quality open datasets and applying cross-deduplication and model-based quality filtering. The dataset comprises diverse sources such as web data, highly educational content, math, code, and scientific papers. Zyda-2 is constructed based on the best open-source datasets, including Zyda, FineWeb, DCLM, and Dolma. Models trained on Zyda-2 significantly outperform identical models trained on the Pile, RefinedWeb, FineWeb, FineWeb-Edu, and DCLM. Due to its post-processing deduplication, filtering, and weighting pipeline, Zyda-2 outperforms all its constituent datasets in model quality.

搜集汇总

背景与挑战

背景概述

Zyda-2是一个规模超过1T、包含5万亿token的英语文本生成数据集，由多个高质量开源数据集经过交叉去重和模型过滤组合而成，涵盖网页数据、教育内容、数学、代码和科学论文等多种来源。该数据集在模型训练中表现出色，据称是每token性能最佳的开源数据集之一，尤其擅长教育内容和自然语言推理。