finepdfs-100M

## Sampling Methodology

This dataset was created using **reservoir sampling**, a statistically unbiased random sampling algorithm that guarantees each sample from the source dataset has an equal probability of being included. This ensures the 100M token sample is representative of the full dataset's characteristics.

**Source Dataset**: [HuggingFaceFW/finepdfs](https://huggingface.co/datasets/HuggingFaceFW/finepdfs)
**Sample Size**: 100M tokens
**Content**: High-quality textbook-style pdfs

Reservoir sampling enables rapid experimentation and ablation studies without processing the entire source dataset, while maintaining statistical validity of results.

For details on how this dataset was used in optimal pre-training data composition research, see the [blog post](https://huggingface.co/blog/codelion/optimal-dataset-mixing/).

## Citation

If you use this model/dataset, please cite:

```bibtex
@article{sharma2025billion,
title={The 1 Billion Token Challenge: Finding the Perfect Pre-training Mix},
author={Sharma, Asankhaya},
year={2025},
url={https://huggingface.co/blog/codelion/optimal-dataset-mixing/}
}
```

For more details, see the [blog post](https://huggingface.co/blog/codelion/optimal-dataset-mixing/).

# 采样方法
本数据集采用**蓄水池采样（reservoir sampling）**方法构建，该算法为统计无偏随机采样算法，可确保源数据集中的每个样本被选中的概率均等。这可保证该1亿Token（Token）样本能够反映完整数据集的整体特征。

**源数据集**：[HuggingFaceFW/finepdfs](https://huggingface.co/datasets/HuggingFaceFW/finepdfs)
**采样规模**：1亿Token
**内容**：高质量教科书风格的PDF文档

蓄水池采样允许在无需处理完整源数据集的前提下开展快速实验与消融研究，同时保障实验结果的统计有效性。

若需了解该数据集在最优预训练数据配比研究中的具体应用细节，请参阅[博客文章](https://huggingface.co/blog/codelion/optimal-dataset-mixing/)。

# 引用说明
若您使用本模型或数据集，请引用如下文献：
bibtex
@article{sharma2025billion,
title={The 1 Billion Token Challenge: Finding the Perfect Pre-training Mix},
author={Sharma, Asankhaya},
year={2025},
url={https://huggingface.co/blog/codelion/optimal-dataset-mixing/}
}

更多详情请参阅[博客文章](https://huggingface.co/blog/codelion/optimal-dataset-mixing/)。