UW/olmo-mix-1124-subset-p99

这是一个从allenai/olmo-mix-1124数据集中随机抽取的文档子集，其中对极端长度的文档进行了截断处理，以确保数据集的长度分布更加均匀。该数据集包括训练集和评估集，用于token器的训练和编码效率评估。This dataset is a subset of documents sampled uniformly at random from allenai/olmo-mix-1124, with truncation applied to the longest 1% of documents to ensure a more even distribution of document lengths. It includes both a training set and an evaluation set for tokenizer training and encoding efficiency assessment.