Na0s/Next_Token_Prediction_dataset

该数据集主要包含英文文本数据，具有多个特征，包括文本内容、元数据（如redpajama_set_name）、索引、输入ID、注意力掩码和标签。数据集分为训练集和验证集，训练集包含5489000个样本，验证集包含9347个样本。数据集的下载大小为17538263800字节，总大小为33172457338字节。

This dataset primarily contains English text data with multiple features, including text content, metadata (such as redpajama_set_name), index, input IDs, attention masks, and labels. The dataset is divided into a training set and a validation set, with the training set containing 5,489,000 samples and the validation set containing 9,347 samples. The download size of the dataset is 17,538,263,800 bytes, and the total size is 33,172,457,338 bytes.