haritzpuerto/the_pile_arxiv_50k_sample

该数据集包含文本数据和元数据，其中文本数据存储在text字段中，元数据存储在meta字段中，包含pile_set_name信息。数据集分为训练集、验证集和测试集，分别包含50000、2434和2407个样本。数据集的下载大小为1227227202字节，总大小为2642500135字节。数据文件的路径和分割方式在配置部分有详细说明。

This dataset contains text data and metadata, with text data stored in the text field and metadata stored in the meta field, which includes pile_set_name information. The dataset is divided into training, validation, and test sets, containing 50,000, 2,434, and 2,407 samples respectively. The download size of the dataset is 1,227,227,202 bytes, and the total size is 2,642,500,135 bytes. The configuration section details the paths and splits of the data files.