Ba2han/merged_pt_2404

Ba2han/merged_pt_2404 是一个合并的预训练语料库，于2026年4月25日从7个源数据集组装而成。该数据集仅包含input_ids列，并转换为uint16数据类型以提高存储效率，但需注意uint16的最大值为65535，超过此阈值的令牌ID会溢出。数据集总令牌数为354.1156亿（35,411,560,020），分为473个Parquet分片（每个分片约7487万令牌），使用Zstandard压缩格式。源数据集包括tokenized-corpus-0903、warmup_turmix、mixed_corpus-2303、English_mix-tokenized、forumsoh_tokenized、3.5b-tokens和long-corpus，覆盖土耳其语和英语，主要用于文本生成和预训练任务。数据集的许可证继承自源数据集，需检查各源仓库的单独许可证。

Ba2han/merged_pt_2404 is a merged pre-training corpus assembled on 2026-04-25 from 7 source datasets. It only stores the input_ids column, cast to uint16 for storage efficiency, with a note that uint16 has a maximum value of 65535 and token IDs above this threshold will overflow. The total token count is 35.41 billion (35,411,560,020), distributed across 473 Parquet shards (approximately 74.87 million tokens per shard) in Zstandard-compressed format. Source datasets include tokenized-corpus-0903, warmup_turmix, mixed_corpus-2303, English_mix-tokenized, forumsoh_tokenized, 3.5b-tokens, and long-corpus, covering Turkish and English languages, and intended for text-generation and pretraining tasks. The license is inherited from the source datasets, and users should check each source repository for its individual license.