mikasenghaas/fineweb-edu-10bt-tokenized
收藏Hugging Face2024-12-29 更新2025-02-15 收录
下载链接：
https://hf-mirror.com/datasets/mikasenghaas/fineweb-edu-10bt-tokenized
下载链接
链接失效反馈官方服务：
资源简介：
该数据集包含用于训练的文本数据，其中包括input_ids和attention_mask两个特征。input_ids是文本的整数表示，attention_mask用于指示序列中的有效位置。数据集分为训练集，共有约967万示例，数据集大小为约49.6GB。
The dataset contains text data for training, which includes two features: input_ids and attention_mask. input_ids are integer representations of the text, and attention_mask is used to indicate the valid positions in the sequence. The dataset is split into a training set with a total of approximately 9.67 million examples, and the dataset size is approximately 49.6GB.
提供机构：
mikasenghaas
