sinatras/fineweb10B-llama3
收藏Hugging Face2024-11-30 更新2025-10-25 收录
下载链接：
https://hf-mirror.com/datasets/sinatras/fineweb10B-llama3
下载链接
链接失效反馈官方服务：
资源简介：
---
license: mit
task_categories:
- text-generation
size_categories:
- 10M<n<100M
---
# FineWeb Dataset tokenized with LLaMA-3.2-1B tokenizer
This dataset contains the FineWeb dataset preprocessed and tokenized using the LLaMA-3.2-1B tokenizer.
Each .bin file contains:
- A header of 256 int32 values
- Token data as uint32 values
- File format version: 2
Dataset structure:
- fineweb_train_*.bin: Training shards
- fineweb_val_*.bin: Validation shard
提供机构：
sinatras
