ltg/saami-web

这个开放语料库是通过对北萨米语网页进行网络爬取的结果。爬取过程从北萨米语维基百科的外部链接开始，使用GlotLID识别北萨米语网页，并遵守Robots Exclusion Protocol。原始HTML文档使用Trafilatura转换为自然文本，并在文档级别进行了模糊去重。许可证对数据集内容没有额外限制。

The Northern Sámi Web Corpus is an open corpus created by crawling admissible web pages in Northern Sámi. The crawl was seeded from the external links of the Sámi Wikipedia and continued with a breadth-first search through webpages identified as Northern Sámi using the GlotLID tool, while adhering to the Robots Exclusion Protocol. The raw HTML documents were converted into natural text using the Trafilatura tool and the corpus has been fuzzy-deduplicated at the document level.