mutiyama/alt|自然语言处理数据集|多语言翻译数据集

亚洲语言树库（ALT）是一个多语言数据集，支持13种亚洲语言，包括孟加拉语、英语、菲律宾语、印地语、印度尼西亚语、日语、高棉语、老挝语、马来语、缅甸语、泰语、越南语和中国语（简体中文）。该数据集旨在通过开放合作推动亚洲自然语言处理技术的发展，适用于机器翻译和依赖解析等任务。ALT由日本国家信息和通信技术研究所（NICT）等机构合作开发，采用专家生成和众包方式进行注释，并根据CC-BY-4.0许可证发布。

原始信息汇总

数据集概述

数据集名称

• Asian Language Treebank (ALT)

语言支持

• 13种语言：Bengali, English, Filipino, Hindi, Bahasa Indonesia, Japanese, Khmer, Lao, Malay, Myanmar (Burmese), Thai, Vietnamese, Chinese (Simplified Chinese)

许可协议

• Creative Commons Attribution 4.0 International (CC BY 4.0)

多语言性

• 多语言
• 翻译

任务类别

• 翻译
• token-classification

数据集大小

• 100K<n<1M
• 10K<n<100K

数据集创建者

• National Institute of Information and Communications Technology, Japan (NICT)
• University of Computer Studies, Yangon, Myanmar (UCSY)
• Badan Pengkajian dan Penerapan Teknologi, Indonesia (BPPT)
• Institute for Infocomm Research, Singapore (I2R)
• Institute of Information Technology, Vietnam (IOIT)
• National Institute of Posts, Telecoms and ICT, Cambodia

数据集配置

• config_name: alt-en, alt-jp, alt-km, alt-my, alt-my-transliteration, alt-my-west-transliteration, alt-parallel

数据集特征

• alt-en
  • features: SNT.URLID, SNT.URLID.SNTID, url, status, value
  • splits: train, validation, test
• alt-jp
  • features: SNT.URLID, SNT.URLID.SNTID, url, status, value, word_alignment, jp_tokenized, en_tokenized
  • splits: train, validation, test
• alt-km
  • features: SNT.URLID, SNT.URLID.SNTID, url, km_pos_tag, km_tokenized
  • splits: train, validation, test
• alt-my
  • features: SNT.URLID, SNT.URLID.SNTID, url, value
  • splits: train, validation, test
• alt-my-transliteration
  • features: en, my
  • splits: train
• alt-my-west-transliteration
  • features: en, my
  • splits: train
• alt-parallel
  • features: SNT.URLID, SNT.URLID.SNTID, url, translation
  • splits: train, validation, test

数据集下载和大小

• alt-en
  • download_size: 3781814
  • dataset_size: 11187560
• alt-jp
  • download_size: 10355366
  • dataset_size: 24245424
• alt-km
  • download_size: 4344096
  • dataset_size: 13344316
• alt-my
  • download_size: 6569025
  • dataset_size: 22679830
• alt-my-transliteration
  • download_size: 2163951
  • dataset_size: 4249316
• alt-my-west-transliteration
  • download_size: 2857511
  • dataset_size: 7411911
• alt-parallel
  • download_size: 34707907
  • dataset_size: 75971326

引用信息

@inproceedings{riza2016introduction, title={Introduction of the asian language treebank}, author={Riza, Hammam and Purwoadi, Michael and Uliniansyah, Teduh and Ti, Aw Ai and Aljunied, Sharifah Mahani and Mai, Luong Chi and Thang, Vu Tat and Thai, Nguyen Phuong and Chea, Vichet and Sam, Sethserey and others}, booktitle={2016 Conference of The Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques (O-COCOSDA)}, pages={1--6}, year={2016}, organization={IEEE} }