MK0727/CleanedWiki-jp
收藏资源简介：
CleanedWiki-jp是一个为LLM预训练准备的清洗后的日语维基百科数据集。它基于日语维基百科文章的HTML构建，转换为Markdown格式，并经过过滤以提高可训练性。数据集保留了有用的文章结构，而不是将所有内容扁平化为纯文本。合适的正文表格被保留为Markdown表格，数学表达式以TeX形式保留。每行还包括预测的日本十进制分类（NDC）类别和jReadability难度级别，允许用户按领域和阅读难度重新平衡训练混合。
CleanedWiki-jp is a cleaned Japanese Wikipedia dataset prepared for LLM pre-training. It is built from Japanese Wikipedia article HTML, converted into Markdown, filtered for trainability. The dataset keeps useful article structure instead of flattening everything into plain text. Suitable body tables are preserved as Markdown tables, and mathematical expressions are preserved in TeX form. Each row also includes a predicted Nippon Decimal Classification (NDC) category and a jReadability difficulty level, allowing users to rebalance training mixtures by domain and reading difficulty.
