chinese-c4

## Introduction

Chinese-C4 is a clean Chinese internet dataset based on Common Crawl. The dataset is 46.29GB and has undergone multiple cleaning strategies, including Chinese filtering, heuristic cleaning based on punctuation, line-based hashing for deduplication, and repetition removal.

The dataset is open source and free for commercial use, and you are welcome to use the data and the cleaning strategies provided and contribute your cleaning strategies.

You can find the cleaning script for the dataset on GitHub [c4-dataset-script](https://github.com/shjwudp/c4-dataset-script).

## 简介
Chinese-C4是一款基于通用爬虫语料库（Common Crawl）构建的纯净中文互联网数据集，数据集规模为46.29GB，已通过多种清洗策略完成预处理，涵盖中文文本过滤、基于标点的启发式清洗、基于行哈希的去重以及重复内容移除。
本数据集开源且可免费商用，欢迎使用者复用本数据集与官方提供的清洗策略，并贡献自研的清洗方案。
您可在GitHub平台的[c4-dataset-script](https://github.com/shjwudp/c4-dataset-script)仓库中获取本数据集的清洗脚本。

搜集汇总

背景与挑战

背景概述

Chinese-C4是一个基于Common Crawl的干净中文互联网数据集，大小为46.29GB，经过多种清理策略处理，包括中文过滤、去重和重复移除。该数据集开源且免费用于商业用途，适用于中文自然语言处理任务。