trixyL/simplestories-8k-megatron

--- license: apache-2.0 task_categories: - text-generation language: - en tags: - megatron - 8k - megadlms pretty_name: Megatron 8k SimpleStories size_categories: - 1M<n<10M --- <h1 align="center">📦 Megatron-LM/MegaDLMs Preprocessed Dataset</h1> This **dataset** hosts the Megatron-LM/MegaDLMs preprocessed SimpleStories dataset using an 8k vocab BPE Tokenizer. ## ✅ What this contains - Preprocessed Megatron dataset files (e.g., `.bin` / `.idx`) ready for Megatron-LM/MegaDLMs training - BPE Tokenizer config files used to create the dataset ## 🔗 References Fork with extra preprocessing utils for SimpleStories: https://github.com/triloy8/MegaDLMs Original MegaDLMs repo: https://github.com/JinjieNi/MegaDLMs SimpleStories dataset: https://huggingface.co/datasets/SimpleStories/SimpleStorieslicense: apache-2.0 task_categories: - 文本生成 language: - 英语 tags: - Megatron - 8k - MegaDLMs pretty_name: Megatron 8k SimpleStories size_categories: - 100万 < 样本量 < 1000万 --- <h1 align="center">📦 Megatron-LM/MegaDLMs 预处理数据集</h1> 本**数据集**针对SimpleStories数据集完成预处理，采用8k词表的字节对编码（Byte Pair Encoding，BPE）分词器生成适配Megatron-LM与MegaDLMs框架的训练数据。 ## ✅ 数据集包含内容 - 预处理完成的Megatron格式数据集文件（例如`.bin`与`.idx`格式），可直接用于Megatron-LM/MegaDLMs的模型训练 - 构建本数据集所用的BPE分词器配置文件 ## 🔗 参考资料 针对SimpleStories的额外预处理工具分支仓库： https://github.com/triloy8/MegaDLMs MegaDLMs官方原始仓库： https://github.com/JinjieNi/MegaDLMs SimpleStories数据集来源： https://huggingface.co/datasets/SimpleStories/SimpleStories