JakeOh/fineweb-edu

该数据集包含三个配置：sample-10BT-split、sample-10BT-tokenized和sample-10BT-tokenized-split。每个配置都有特定的特征和划分为训练集与测试集。sample-10BT-split配置的特征包括文本、ID、转储、URL、文件路径、语言、语言分数、词汇计数、分数和整分数。而tokenized配置的特征包括输入ID、注意力掩码和特殊令牌掩码。数据集的大小和下载大小也有说明。

The dataset consists of three configurations: sample-10BT-split, sample-10BT-tokenized, and sample-10BT-tokenized-split. Each configuration has specific features and is split into training and testing sets. The sample-10BT-split configuration includes features such as text, id, dump, url, file_path, language, language_score, token_count, score, and int_score. The tokenized configurations feature input_ids, attention_mask, and special_tokens_mask. The dataset size and download size are also specified.