agentlans/fineweb2-chinese

FineWeb2 - Chinese 是一个中文文本数据集，源自FineWeb2数据集的cmn_Hani子集。该数据集按地区或类别划分，包括中国大陆、台湾、香港、模糊的传统中文（可能来自台湾或香港）以及其他地区的文本。数据以JSONL ZST压缩格式提供，总行数统计显示：中国大陆部分有834,356行，台湾有83,875行，香港有10,411行，模糊的传统中文有67,700行，其他地区有3,658行。数据集可能包含简体和繁体中文文本，适用于自然语言处理任务，如文本分析、语言模型训练等。

FineWeb2 - Chinese is a Chinese text dataset derived from the cmn_Hani subset of the FineWeb2 dataset. It is organized by region or category, including texts from Mainland China, Taiwan, Hong Kong, ambiguous traditional Chinese (likely from Taiwan or Hong Kong), and other regions. The data is provided in JSONL ZST compressed format, with row counts as follows: Mainland China has 834,356 rows, Taiwan has 83,875 rows, Hong Kong has 10,411 rows, ambiguous traditional Chinese has 67,700 rows, and other regions have 3,658 rows. The dataset may contain both simplified and traditional Chinese texts, suitable for natural language processing tasks such as text analysis and language model training.