EusCrawl

EusCrawl是由HiTZ中心和巴斯克大学创建的高质量巴斯克语数据集，包含12.5M文档和423M令牌，主要通过手动识别和抓取高质量内容的网站构建。该数据集在质量上显著优于其他多语言数据集，如mC4和CC100，由本地注释者评估。EusCrawl的应用领域包括自然语言理解任务，旨在解决低资源语言在NLP中的性能问题，尤其是在数据量和领域覆盖方面的重要性。

EusCrawl is a high-quality Basque language dataset developed by the HiTZ Center and the University of the Basque Country. It contains 12.5 million documents and 423 million tokens, and was primarily constructed by manually identifying and scraping websites with high-quality content. Evaluated by native annotators, this dataset significantly outperforms other multilingual datasets such as mC4 and CC100 in terms of quality. EusCrawl targets natural language understanding tasks, aiming to address the performance issues of low-resource languages in NLP, particularly highlighting the importance of data volume and domain coverage.