fineweb-edu-100M

## Sampling Methodology

This dataset was created using **reservoir sampling**, a statistically unbiased random sampling algorithm that guarantees each sample from the source dataset has an equal probability of being included. This ensures the 100M token sample is representative of the full dataset's characteristics.

**Source Dataset**: [HuggingFaceFW/fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)
**Sample Size**: 100M tokens
**Content**: Curated educational web resources

Reservoir sampling enables rapid experimentation and ablation studies without processing the entire source dataset, while maintaining statistical validity of results.

For details on how this dataset was used in optimal pre-training data composition research, see the [blog post](https://huggingface.co/blog/codelion/optimal-dataset-mixing/).

## Citation

If you use this model/dataset, please cite:

```bibtex
@article{sharma2025billion,
title={The 1 Billion Token Challenge: Finding the Perfect Pre-training Mix},
author={Sharma, Asankhaya},
year={2025},
url={https://huggingface.co/blog/codelion/optimal-dataset-mixing/}
}
```

For more details, see the [blog post](https://huggingface.co/blog/codelion/optimal-dataset-mixing/).

# 采样方法
本数据集采用**蓄水池采样（reservoir sampling）**构建，这是一种统计无偏的随机采样算法，可确保源数据集中的每个样本被选中的概率均等。此举可保证该1亿Token（Token）样本能够反映完整数据集的整体特征。

**源数据集**：[HuggingFaceFW/fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)
**采样规模**：1亿Token（Token）
**内容**：精选教育类网络资源

蓄水池采样可在无需处理完整源数据集的前提下，支持快速实验与消融研究，同时保证实验结果的统计有效性。

如需了解该数据集在最优预训练数据组合研究中的具体应用方式，请参阅此[博客文章](https://huggingface.co/blog/codelion/optimal-dataset-mixing/)。

# 引用方式
若您使用此模型或数据集，请引用如下文献：
bibtex
@article{sharma2025billion,
title={《10亿Token挑战：探寻最优预训练数据组合》},
author={夏尔马, 阿桑卡亚（Sharma, Asankhaya）},
year={2025},
url={https://huggingface.co/blog/codelion/optimal-dataset-mixing/}
}

如需了解更多细节，请参阅此[博客文章](https://huggingface.co/blog/codelion/optimal-dataset-mixing/)。