meryyllebr543/ultrafineweb-100B-tokens
收藏官方服务：
资源简介：
Tokenized Ultra-FineWeb是一个包含1000亿个高质量英语token的数据集，专为大规模语言模型训练准备。数据集经过自定义BPE tokenizer处理，并以分片形式存储，共有100个文件，每个文件包含10亿个token，便于流式读取和分布式训练。
Tokenized Ultra-FineWeb is a dataset containing 100 billion high-quality English tokens, prepared for large-scale language model training. The dataset has been processed with a custom BPE tokenizer and is stored in a sharded format, with 100 files each containing 1 billion tokens for easy streaming and distributed training.
提供机构：
meryyllebr543
