HPLT语言资源

HPLT语言资源是一个大规模多语言数据集，包含单语和双语语料，从CommonCrawl和互联网档案馆的未使用网络爬虫中提取。该数据集专注于中低资源语言，覆盖75种语言，总计约5.6万亿词条。英语中心平行语料库涵盖18种语言对和超过9600万对对齐句子，约含14亿英语词条。HPLT语言资源是迄今为止发布的最大开放文本语料库之一，为语言建模和机器翻译训练提供了丰富的资源。数据集的创建过程依赖于开源软件工具和高性能计算，其应用领域旨在解决语言技术中的资源稀缺问题，推动多语言技术的发展。

The HPLT Language Resource is a large-scale multilingual dataset containing monolingual and bilingual corpora, extracted from unused web crawls of CommonCrawl and the Internet Archive. This dataset focuses on low- and medium-resource languages, covering 75 languages with a total of approximately 5.6 trillion tokens. The English-centric parallel corpus covers 18 language pairs and over 96 million aligned sentence pairs, containing approximately 1.4 billion English tokens. The HPLT Language Resource is one of the largest open text corpora released to date, providing abundant resources for language modeling and machine translation training. The dataset's creation process relies on open-source software tools and high-performance computing, with its application scenarios aiming to address the resource scarcity issue in language technology and promote the development of multilingual technologies.