gair-prox/FineWeb-pro

fineweb-pro是从fineweb（350BT样本）中精炼出来的，使用了ProX精炼框架。它包含了大约100B的高质量token，适合用于通用语言模型的预训练。数据集基于ODC-By 1.0许可证，用户还需遵守CommonCrawl的使用条款。fineweb-pro is refined from fineweb (350BT sample) using the ProX refining framework. It contains about 100B high quality tokens, ready for general language model pre-training. The dataset is based on the ODC-By 1.0 license, and users should also abide by the CommonCrawl Terms of Use.