WanJuan-CC

万卷-CC(WanJuan-CC) 是从CommonCrawl获取的一个 1T Tokens 的高质量英文网络文本数据集。结果显示，与各类开源英文CC语料在 Perspective API 不同维度的评估上，WanJuan-CC都表现出更高的安全性。此外，通过在4个验证集上的困惑度（PPL）和6下游任务的准确率，也展示了WanJuan-CC的实用性。WanJuan-CC在各种验证集上的PPL表现出竞争力，特别是在要求更高语言流畅性的tiny-storys等集上。通过与同类型数据集进行1B模型训练对比，使用验证数据集的困惑度（perplexity）和下游任务的准确率作为评估指标，实验证明，WanJuan-CC显著提升了英文文本补全和通用英文能力任务的性能。WanJuan-CC is a high-quality English web text dataset with 1T Tokens, sourced from CommonCrawl. Experimental results demonstrate that WanJuan-CC exhibits higher safety performance compared with various open-source English CC corpora when evaluated across different dimensions via Perspective API. Furthermore, its practical utility is validated by perplexity (PPL) scores on 4 validation datasets and accuracy metrics across 6 downstream tasks. WanJuan-CC shows competitive PPL performance across all tested validation datasets, particularly on datasets like TinyStories that demand higher linguistic fluency. Through comparative training of a 1B-parameter model using analogous datasets, with perplexity (PPL) on validation datasets and accuracy on downstream tasks serving as evaluation metrics, experiments prove that WanJuan-CC significantly improves the performance of English text completion and general English proficiency tasks.

搜集汇总

背景与挑战

背景概述

WanJuan-CC是一个从CommonCrawl获取的高质量英文网络文本数据集，数据量约为1T Tokens。该数据集通过多步骤数据处理确保高质量和安全性，在Perspective API评估中表现更安全，并在验证集困惑度和下游任务准确率上显示其实用性，能显著提升英文文本补全和通用英文能力任务性能。