blester125/wiki-dolma

Wiki数据集是通过wikiteam工具收集的开放许可的wiki转储，并进行了预处理。数据集包括多个版本，每个版本都有不同的预处理步骤，如从原始wikitext到去除非开放许可内容的多个版本。数据集中的文档可能包含重复内容，建议在使用前应用模糊去重过滤器。

This dataset contains preprocessed versions of openly licensed wiki dumps collected by wikiteam and hosted on the Internet Archive. The dataset has multiple versions, each with different processing steps such as parsing wikitext to plain text, removing HTML snippets, and removing non-openly licensed content.