Ba2han/merged_pretraining_2704

Merged Pretraining 2704是一个用于文本生成任务的标记化预训练数据集。该数据集由多个源数据集合并而成，每个源数据集都贡献了大量行数据。数据以Parquet格式存储，包含一个名为input_ids的列，该列由无符号32位整数列表组成。数据集通过流式处理每个源数据集、验证标记ID序列、规范化标记ID并合并所有源数据行的方式处理。最终数据集包含67,937,789行数据，分布在272个分片中，除最后一个分片外，每个分片包含约250,000行数据。Merged Pretraining 2704 is a tokenized pretraining dataset for text-generation tasks. The dataset is a merge of multiple source datasets, each contributing a significant number of rows. The data is stored in Parquet format with a single column input_ids containing lists of unsigned 32-bit integers. The dataset has been processed by streaming each source dataset, validating token-id sequences, normalizing token IDs, and merging rows from all sources. The final dataset contains 67,937,789 rows distributed across 272 shards, with each shard containing approximately 250,000 rows except the last one.