EleutherAI/fineweb-edu-dedup-10b

该数据集包含文本数据及其相关元数据，文本数据包含标识符和内容，元数据包括dump信息、URL、日期、文件路径、语言、语言评分、词数、评分和整数量评分等。数据集划分为训练集，提供了训练集的字节数和示例数，以及数据集的下载大小和整体大小。

The dataset consists of text data and its associated metadata, including identifiers and content. The metadata includes dump information, URL, date, file path, language, language score, token count, score, and integer score. The dataset is split into a training set, with the number of bytes and examples provided for the training set, as well as the download size and total size of the dataset.