jack-stanley/fineweb-edu-dedup-10b-10gram-shuffled

该数据集包含文本内容和相关元数据，文本内容字段名为text，每条记录都有一个唯一标识符id。元数据(metadata)包含.dump、url、日期(date)、文件路径(file_path)、语言(language)、语言置信度(language_score)、词汇数量(token_count)、得分(score)和整数值得分(int_score)等信息。数据集被划分为训练集(train)，大小为48115109854字节，共有9508400个示例。数据集的配置信息包括默认配置，指定了训练集的数据文件路径。

The dataset includes text content and associated metadata, with the text field named text and each record having a unique identifier id. The metadata contains .dump, url, date, file_path, language, language_score, token_count, score, and int_score. The dataset is split into a training set (train), which is 48115109854 bytes in size and contains 9508400 examples. The configuration information for the dataset includes a default configuration that specifies the data file path for the training set.