pico-lm/pretokenized-paloma

Pretokenized Paloma是一个紧凑的、预标记化的评估数据集，旨在补充pretokenized-dolma训练集。它由Paloma语料库构建而成，与Dolma数据集没有数据重叠，适合用于在pretokenized-dolma上训练的模型的困惑度计算。

The Pretokenized Paloma Benchmark Dataset is a compact, pre-tokenized evaluation dataset designed to complement the pretokenized-dolma training set. Built from the Paloma corpus (Allen Institute), this benchmark is free from any data overlap with Dolma and is ideal for evaluating models trained on it for perplexity calculations.