marin-community/wikipedia-markdown

Marin Markdownified Wikipedia是一个大规模的、经过预处理的英语维基百科企业HTML转储版本，包含约85.9亿个标记。这个语料库已经被转换为干净的、具有章节意识的Markdown格式，用于语言模型训练。Marin Markdownified Wikipedia is a large-scale, pre-processed version of the English Wikipedia Enterprise HTML dump consisting of approximately 8.59 billion tokens. This corpus has been converted into a clean, section-aware Markdown format for language-model training.