javiersgjavi/fineweb-1BT

FineWeb-1BT是一个经过精心策划的10亿token子集，从HuggingFaceFW/fineweb数据集中独立采样而来，确保在整个10亿token语料库中的无偏代表。这个子集使用真正的随机采样方法创建，以保持原FineWeb数据集的语言/质量特性。FineWeb-1BT is a carefully curated 1 billion token subset of the HuggingFaceFW/fineweb dataset, sampled exclusively from the official 10BT FineWeb subset, ensuring unbiased representation across the entire 10BT corpus. This subset was created using true random sampling to preserve the language/quality characteristics of the original FineWeb dataset.