lumasik/wiki-simple-cleaned

Wiki-Simple-Cleaned数据集是一个深度清理的Simple English Wikipedia版本，基于legacy-datasets/wikipedia。该数据集专门设计用于预训练微模型语言架构，旨在移除Wiki数据转储中的噪声（如服务部分、损坏的标记、参考文献列表），使模型能够专注于纯语言和事实学习，而非网页结构。清理流程包括：删除参考文献、参见、外部链接、延伸阅读和来源等部分；将维基标题（例如==历史==）转换为带句点的普通文本句子以保持叙事逻辑；格式化文本，如将所有换行符替换为空格、删除多余空格和空行，并将每篇文章打包成单行；每个文档用开始和结束序列标记包装，格式为<|bos|>{标题}。{清理后的文本}<|eos|>。数据集格式为文本文件（.txt），每行代表一个文档，语言为简单英语，适合训练参数不超过5亿的模型。分布统计显示：约70%为短文章（摘要），约27%为中等文章，约3%为长篇文章。使用建议包括采用序列打包技术，将多篇文章拼接至单个上下文窗口，以最大化GPU显存利用率。

Wiki-Simple-Cleaned is a deeply cleaned version of Simple English Wikipedia, based on legacy-datasets/wikipedia. The dataset is specifically prepared for pre-training micro-model language architectures. The main goal is to remove noise typical of Wiki dumps (such as service sections, broken markup, and reference lists), allowing the model to focus on pure language and facts rather than learning web page structure. The cleaning pipeline includes: removing tails like References, See also, External links, Further reading, and Sources; converting wiki headings (e.g., == History ==) into regular text sentences with periods to preserve narrative logic; formatting text by replacing all line breaks with spaces, removing multiple spaces and empty lines, and packing each article into a single line; and wrapping each document with special tokens: <|bos|>{Title}. {Cleaned Text}<|eos|>. The format is a plain text file (.txt) where each line represents one document, in English (Simple) language, ideal for training models with up to 500M parameters. Distribution statistics show approximately 70% short articles (summaries), 27% medium articles, and 3% long articles. Usage recommendations include employing Sequence Packing to concatenate multiple articles into a single context window for efficient GPU VRAM utilization.