snimu/fineweb-edu-sample-10BT-tiktokenized
收藏Hugging Face2024-08-17 更新2024-12-14 收录
下载链接：
https://hf-mirror.com/datasets/snimu/fineweb-edu-sample-10BT-tiktokenized
下载链接
链接失效反馈官方服务：
资源简介：
该数据集配置名为gpt2，包含多个特征如tokens（序列类型，int64）、token_count（int64）、id（字符串）、dump（字符串）、url（字符串）、file_path（字符串）、language（字符串）、language_score（float64）、score（float64）和int_score（int64）。数据集分为训练集和验证集，训练集包含9,671,973个样本，验证集包含128个样本。数据集的下载大小为20,839,487,016字节，总大小为82,678,436,550字节。
The dataset, configured under the name gpt2, includes multiple features such as tokens (sequence, int64), token_count (int64), id (string), dump (string), url (string), file_path (string), language (string), language_score (float64), score (float64), and int_score (int64). It is divided into training and validation sets, with the training set containing 9,671,973 samples and the validation set containing 128 samples. The download size of the dataset is 20,839,487,016 bytes, and the total size is 82,678,436,550 bytes.
提供机构：
snimu
