ivykopal/fineweb2-slovak

这是FineWeb2数据集的斯洛伐克语部分，包含超过141亿个单词，跨越超过2650万份文档。该数据集通过FastText、langdetect和lingua等工具增加了语言识别功能，用于筛选出非斯洛伐克语的文本。此外，数据集还包含了识别出的成人词汇列表，可用于过滤带有问题内容文本。

This is the Slovak portion of the FineWeb2 dataset, containing over 14.1 billion words across more than 26.5 million documents. The dataset has been enhanced with language identification features using FastText, langdetect, and lingua, to filter out non-Slovak texts. Additionally, it includes a list of identified adult words (both Slovak and English) for filtering out texts with problematic content.