ChineseWebText

该数据集是从CommonCrawl数据中提取的单语种中文质量数据集，根据人工制定的规则进行了筛选。在筛选过程中，考虑了数据长度、敏感词汇以及中文字符的比例。该数据集的规模达到了530吉字节，其任务旨在为机器翻译进行预训练。

This dataset is a high-quality monolingual Chinese dataset extracted from CommonCrawl data, filtered based on manually formulated rules. During the filtering process, factors including data length, sensitive vocabulary, and the proportion of Chinese characters were taken into account. The dataset has a total size of 530 gigabytes, and it is intended for pre-training in machine translation tasks.