Geralt-Targaryen/CC-News

English CC-News 2016-2021是一个经过清洗、去重和净化处理的英文新闻数据集，包含了从2016年到2021年的数据。清洗过程去除了非英文内容和低质量内容，如广告。数据集内部进行了全局去重，并与OpenWebText2数据集进行了跨去重。此外，数据集还基于n-gram重叠与多个基准数据集进行了净化处理。该数据集共有约1.33亿个样本，下载的parquet文件大小为221G。

The English CC-News 2016-2021 dataset is a cleaned, deduplicated, and decontaminated English news dataset covering the period from 2016 to 2021. The cleaning process removed non-English content and low-quality content such as advertisements. Global deduplication was performed within the dataset, and cross-deduplication was carried out with the OpenWebText2 dataset. The dataset was also decontaminated based on n-gram overlap with multiple benchmark datasets. The dataset contains approximately 133 million samples, with the downloaded parquet files totaling 221G in size.