mikezhu/fineweb-edu-100B-gpt2-tokenized-1024

FineWeb-Edu 100B数据集是一个预分词版本的数据集，基于原始数据集HuggingFaceFW/fineweb-edu的sample-100BT配置，专门为自回归语言模型预训练而准备。该数据集仅包含训练分割，使用GPT-2分词器进行分词，序列长度固定为1024个标记，文档被连接并分块成固定长度序列，文档间插入了EOS（序列结束）标记。数据集包含97,961,640个序列，约1003亿个标记，存储为1607个Arrow文件，每个文件约500MB。数据模式包括input_ids（int32序列）和attention_mask（float32序列，全为1.0，表示无填充）。该数据集适用于大规模语言模型训练，并遵循ODC-By 1.0许可协议。

FineWeb-Edu 100B is a pre-tokenized version of the original dataset HuggingFaceFW/fineweb-edu (specifically the sample-100BT configuration), prepared for autoregressive language model pretraining. This dataset includes only the training split, with validation typically handled by wikitext-103 (not included). It is tokenized using the GPT-2 tokenizer, with a fixed sequence length of 1024 tokens, where documents are concatenated and chunked into fixed-length sequences, and an EOS token is inserted between documents. The dataset contains 97,961,640 sequences, approximately 100.3 billion tokens, stored in 1607 Arrow files (each ~500 MB). The schema includes input_ids (Sequence of int32) and attention_mask (Sequence of float32, all ones due to wrapping with no padding). It is designed for efficient loading and training of large language models, and is licensed under ODC-By 1.0.