chicane/wikipedia

维基媒体维基百科数据集包含所有语言的清理文章。该数据集是从维基百科转储（https://dumps.wikimedia.org/）构建的，每种语言一个子集，每个子集包含一个训练分割。每个示例包含一篇完整的维基百科文章内容，经过清理以去除标记和不需要的部分（如参考文献等）。所有语言子集都已针对最近的转储进行了处理，可以按日期和语言加载。数据集主要用于语言建模任务。

Wikipedia dataset containing cleaned articles of all languages. The dataset is built from the Wikipedia dumps (https://dumps.wikimedia.org/) with one subset per language, each containing a single train split. Each example contains the content of one full Wikipedia article with cleaning to strip markdown and unwanted sections (references, etc.). All language subsets have already been processed for recent dump, and you can load them per date and language. The dataset is generally used for Language Modeling.