SEACrowd/hplt

Hplt数据集是High Performance Language Technologies项目的一部分，该项目是一个为期三年的欧盟资助项目，始于2022年9月。HPLT从Internet Archive和CommonCrawl中提取单语和双语数据集，构建高效的机器翻译和大语言模型。HPLT旨在利用高性能计算（HPC）提供免费、可持续和可重用的数据集、模型和工作流程。数据集支持自监督预训练任务，语言包括印尼语、马来语、泰语、缅甸语、菲律宾语和越南语。

原始信息汇总

Hplt 数据集概述

基本信息

• 数据集名称: Hplt
• 许可证: Creative Commons Zero v1.0 Universal (cc0-1.0)
• 语言:
  • 印尼语 (ind)
  • 马来语 (zlm)
  • 泰语 (tha)
  • 缅甸语 (mya)
  • 菲律宾语 (fil)
  • 越南语 (vie)
• 任务类别: 自监督预训练 (Self Supervised Pretraining)
• 标签: 自监督预训练 (Self Supervised Pretraining)

数据集版本

• 源版本: 1.2.0
• SEACrowd 版本: 2024.06.20

数据集使用

使用 datasets 库

python from datasets import load_dataset dset = datasets.load_dataset("SEACrowd/hplt", trust_remote_code=True)

使用 seacrowd 库

python import seacrowd as sc

使用默认配置加载数据集

dset = sc.load_dataset("hplt", schema="seacrowd")

检查数据集的所有可用子集（配置名称）

print(sc.available_config_names("hplt"))

使用特定配置加载数据集

dset = sc.load_dataset_by_config_name(config_name="<config_name>")

数据集主页

• 链接: https://hplt-project.org/datasets/v1.2

引用

@inproceedings{aulamo-etal-2023-hplt, title = "{HPLT}: High Performance Language Technologies", author = {Aulamo, Mikko and Bogoychev, Nikolay and Ji, Shaoxiong and Nail, Graeme and Ram{\i}rez-S{a}nchez, Gema and Tiedemann, J{"o}rg and van der Linde, Jelmer and Zaragoza, Jaume}, editor = "Nurminen, Mary and Brenner, Judith and Koponen, Maarit and Latomaa, Sirkku and Mikhailov, Mikhail and Schierl, Frederike and Ranasinghe, Tharindu and Vanmassenhove, Eva and Vidal, Sergi Alvarez and Aranberri, Nora and Nunziatini, Mara and Escart{\i}n, Carla Parra and Forcada, Mikel and Popovic, Maja and Scarton, Carolina and Moniz, Helena", booktitle = "Proceedings of the 24th Annual Conference of the European Association for Machine Translation", month = jun, year = "2023", address = "Tampere, Finland", publisher = "European Association for Machine Translation", url = "https://aclanthology.org/2023.eamt-1.61", pages = "517--518", }

@article{lovenia2024seacrowd, title={SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages}, author={Holy Lovenia and Rahmad Mahendra and Salsabil Maulana Akbar and Lester James V. Miranda and Jennifer Santoso and Elyanah Aco and Akhdan Fadhilah and Jonibek Mansurov and Joseph Marvin Imperial and Onno P. Kampman and Joel Ruben Antony Moniz and Muhammad Ravi Shulthan Habibi and Frederikus Hudi and Railey Montalan and Ryan Ignatius and Joanito Agili Lopo and William Nixon and Börje F. Karlsson and James Jaya and Ryandito Diandaru and Yuze Gao and Patrick Amadeus and Bin Wang and Jan Christian Blaise Cruz and Chenxi Whitehouse and Ivan Halim Parmonangan and Maria Khelli and Wenyu Zhang and Lucky Susanto and Reynard Adha Ryanda and Sonny Lazuardi Hermawan and Dan John Velasco and Muhammad Dehan Al Kautsar and Willy Fitra Hendria and Yasmin Moslem and Noah Flynn and Muhammad Farid Adilazuarda and Haochen Li and Johanes Lee and R. Damanhuri and Shuo Sun and Muhammad Reza Qorib and Amirbek Djanibekov and Wei Qi Leong and Quyet V. Do and Niklas Muennighoff and Tanrada Pansuwan and Ilham Firdausi Putra and Yan Xu and Ngee Chia Tai and Ayu Purwarianti and Sebastian Ruder and William Tjhi and Peerat Limkonchotiwat and Alham Fikri Aji and Sedrick Keh and Genta Indra Winata and Ruochen Zhang and Fajri Koto and Zheng-Xin Yong and Samuel Cahyawijaya}, year={2024}, eprint={2406.10118}, journal={arXiv preprint arXiv: 2406.10118} }