skymizer/pretraining-50B-tokenized-padded

该数据集包含文本数据，每个样本包括文本内容、标记计数和输入ID序列。数据集分为训练集和测试集，训练集包含57,251,823个样本，测试集包含1,141个样本。数据集的下载大小为213,983,075,603字节，总大小为397,279,020,164字节。

This dataset contains text data, with each sample including text content, token count, and input ID sequences. The dataset is divided into a training set and a test set, with the training set containing 57,251,823 samples and the test set containing 1,141 samples. The download size of the dataset is 213,983,075,603 bytes, and the total size is 397,279,020,164 bytes.