jack-stanley/fineweb-edu-dedup-10b-6gram-shuffled

这是一个包含文本数据的数据集，具体内容未在README中明确描述。数据集包含文本(text)、唯一标识符(id)和元数据(metadata)。元数据中包含文本来源的URL、日期、文件路径、语言类型、语言评分、词汇计数、分数和整型分数等信息。数据集分为训练集(train)，大小为约48.1GB，包含950万个示例。数据集的总大小也是约48.1GB。

This dataset contains text data, with no specific description provided in the README. It includes features such as text (text), unique identifier (id), and metadata. The metadata contains information such as the URL of the text source, date, file path, language type, language score, token count, score, and integer score. The dataset is split into a training set (train), which is approximately 48.1GB in size and contains 9.5 million examples. The total size of the dataset is also approximately 48.1GB.