dclm-baseline-10M

## Sampling Methodology

This dataset was created using **reservoir sampling**, a statistically unbiased random sampling algorithm that guarantees each sample from the source dataset has an equal probability of being included. This ensures the 10M token sample is representative of the full dataset's characteristics.

**Source Dataset**: [mlfoundations/dclm-baseline-1.0](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0)
**Sample Size**: 10M tokens
**Content**: Filtered, diverse web content

Reservoir sampling enables rapid experimentation and ablation studies without processing the entire source dataset, while maintaining statistical validity of results.

For details on how this dataset was used in optimal pre-training data composition research, see the [blog post](https://huggingface.co/blog/codelion/optimal-dataset-mixing/).

## Citation

If you use this model/dataset, please cite:

```bibtex
@article{sharma2025billion,
title={The 1 Billion Token Challenge: Finding the Perfect Pre-training Mix},
author={Sharma, Asankhaya},
year={2025},
url={https://huggingface.co/blog/codelion/optimal-dataset-mixing/}
}
```

For more details, see the [blog post](https://huggingface.co/blog/codelion/optimal-dataset-mixing/).

# 采样方法
本数据集采用**水库采样（reservoir sampling）**构建，这是一种无偏随机采样算法，可确保源数据集中的每一个样本被选中的概率均等。这可确保该10M Token样本能够精准反映完整数据集的整体特征。

**源数据集**：[mlfoundations/dclm-baseline-1.0](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0)
**采样规模**：10M Token
**内容**：经过筛选的多样化网络文本内容

水库采样可在无需处理完整源数据集的前提下，支持快速开展实验与消融研究，同时确保实验结果具备统计有效性。

若需了解该数据集在最优预训练数据配比研究中的具体应用细节，请参阅[相关博客文章](https://huggingface.co/blog/codelion/optimal-dataset-mixing/)。

# 引用格式
若您使用本模型或数据集，请引用如下文献：
bibtex
@article{sharma2025billion,
title={The 1 Billion Token Challenge: Finding the Perfect Pre-training Mix},
author={Sharma, Asankhaya},
year={2025},
url={https://huggingface.co/blog/codelion/optimal-dataset-mixing/}
}

更多详细信息请参阅上述博客文章。