finepdfs-1B

## Sampling Methodology

This dataset was created using **reservoir sampling**, a statistically unbiased random sampling algorithm that guarantees each sample from the source dataset has an equal probability of being included. This ensures the 1B token sample is representative of the full dataset's characteristics.

**Source Dataset**: [HuggingFaceFW/finepdfs](https://huggingface.co/datasets/HuggingFaceFW/finepdfs)
**Sample Size**: 1B tokens
**Content**: High-quality textbook-style pdfs

Reservoir sampling enables rapid experimentation and ablation studies without processing the entire source dataset, while maintaining statistical validity of results.

For details on how this dataset was used in optimal pre-training data composition research, see the [blog post](https://huggingface.co/blog/codelion/optimal-dataset-mixing/).

## Citation

If you use this model/dataset, please cite:

```bibtex
@article{sharma2025billion,
title={The 1 Billion Token Challenge: Finding the Perfect Pre-training Mix},
author={Sharma, Asankhaya},
year={2025},
url={https://huggingface.co/blog/codelion/optimal-dataset-mixing/}
}
```

For more details, see the [blog post](https://huggingface.co/blog/codelion/optimal-dataset-mixing/).

## 采样策略

本数据集采用**蓄水池采样（reservoir sampling）**构建而成，这是一种统计上无偏的随机采样算法，可确保源数据集中的每个样本被选中的概率均等。此举可保证该10亿Token（Token）样本能够完整反映源数据集的整体特征。

**源数据集**：[HuggingFaceFW/finepdfs](https://huggingface.co/datasets/HuggingFaceFW/finepdfs)
**采样规模**：10亿Token（Token）
**内容范畴**：高质量教科书类PDF文档

蓄水池采样无需处理完整源数据集即可支持快速实验与消融研究，同时保证实验结果的统计有效性。

如需了解该数据集在最优预训练数据配比研究中的具体应用方式，请参阅[博客文章](https://huggingface.co/blog/codelion/optimal-dataset-mixing/)。

## 引用方式

若您在工作中使用本模型/数据集，请引用如下文献：
bibtex
@article{sharma2025billion,
title={The 1 Billion Token Challenge: Finding the Perfect Pre-training Mix},
author={Sharma, Asankhaya},
year={2025},
url={https://huggingface.co/blog/codelion/optimal-dataset-mixing/}
}

如需了解更多细节，请参阅上述[博客文章](https://huggingface.co/blog/codelion/optimal-dataset-mixing/)。