mikasenghaas/fineweb-edu-10bt

该数据集是一个包含文本信息和其他相关元数据的集合，适用于自然语言处理任务。它包含了训练集，共有约967万示例，数据集大小约为49GB。每个示例都包含了文本内容、唯一标识符、可能的URL、文件路径、语言类型、语言置信度分数、词计数、分数和整型分数等字段。

This dataset is a collection of text and associated metadata suitable for natural language processing tasks. It includes a training set with approximately 9.67 million examples, and the dataset size is about 49GB. Each example contains fields such as text content, unique identifier, potential URL, file path, language type, language confidence score, token count, score, and integer score.