TaiMingLu/fineweb-10b-processed

这是一个包含variants和metadata两个特征的变异数据集。variants特征包含原始数据，而metadata特征包含与数据变异相关的多个信息字段，如base、context、copy_index、degenerate、duplicate、is_duplicate_copy、is_duplicate_source、original_index和variety。数据集分为训练集，包含大约440万个示例，总大小为约13.38GB。

This dataset includes two main features: variants and metadata. The variants feature contains the original data, while the metadata feature includes multiple information fields related to data variation, such as base, context, copy_index, degenerate, duplicate, is_duplicate_copy, is_duplicate_source, original_index, and variety. The dataset is split into a training set, containing about 4.4 million examples, with a total size of approximately 13.38GB.