ai4bharat/IN22-Gen

IN22-Gen是一个新创建的综合性基准数据集，用于评估22种印度语言在多领域、多方向并行上下文中的机器翻译性能。该数据集是IN22的通用多领域评估子集，包含1024个句子，涵盖了新闻、娱乐、文化、法律和印度相关主题等多个领域。数据来源于Wikipedia和Web Sources，支持506个翻译方向的评估。数据集结构包括id、context、source、url、domain、num_words、bucket和sentence等字段。

IN22-Gen is a newly created comprehensive benchmark dataset designed to evaluate machine translation performance of 22 Indian languages in multi-domain, multi-directional parallel contexts. This dataset is a general multi-domain evaluation subset of IN22, containing 1024 sentences spanning multiple domains such as news, entertainment, culture, law and India-related topics. The data is sourced from Wikipedia and Web Sources, supporting evaluations across 506 translation directions. The dataset structure includes fields such as id, context, source, url, domain, num_words, bucket and sentence.

原始信息汇总

IN22-Gen 数据集概述

基本信息

• 语言:
  • 包含语言: 阿萨姆语, 孟加拉语, 博多语, 多格里语, 英语, 孔卡尼语, 古吉拉特语, 印地语, 卡纳达语, 克什米尔语, 迈蒂利语, 马拉雅拉姆语, 马拉地语, 曼尼普尔语, 尼泊尔语, 奥里亚语, 旁遮普语, 梵语, 桑塔利语, 信德语, 泰米尔语, 泰卢固语, 乌尔都语
  • 语言详情: asm_Beng, ben_Beng, brx_Deva, doi_Deva, eng_Latn, gom_Deva, guj_Gujr, hin_Deva, kan_Knda, kas_Arab, mai_Deva, mal_Mlym, mar_Deva, mni_Mtei, npi_Deva, ory_Orya, pan_Guru, san_Deva, sat_Olck, snd_Deva, tam_Taml, tel_Telu, urd_Arab
• 许可证: cc-by-4.0
• 语言创建者: 专家生成
• 多语言性: 多语言, 翻译
• 数据集名称: in22-gen
• 数据规模: 1K<n<10K
• 任务类别: 翻译

数据集描述

• 数据集来源: 维基百科和网络资源
• 内容领域: 新闻, 娱乐, 文化, 法律, 印度相关主题
• 句子数量: 1024 句
• 翻译方向: 506 个方向

数据集结构

• 数据字段:
  • id: 数据条目的行号，从1开始。
  • context: 上下文窗口，通常包括候选句子的前后各一句。
  • source: 候选句子的来源。
  • url: 英文文章的URL，仅适用于来自维基百科的候选句子。
  • domain: 句子的领域。
  • num_words: 候选句子中的单词数量。
  • bucket: 根据预定义的桶类别对候选句子进行分类。
  • sentence: 特定语言的完整句子（可能有_lang配对）

数据实例

• 示例: python { "id": 1, "context": "A uniform is often viewed as projecting a positive image of an organisation. Maintaining personal hygiene is also an important aspect of personal appearance and dressing. An appearance is a bunch of attributes related with the service person, like their shoes, clothes, tie, jewellery, hairstyle, make-up, watch, cosmetics, perfume, etc.", "source": "web", "url": "", "domain": "culture", "num_words": 24, "bucket": "18 - 25", "sentence": "An appearance is a bunch of attributes related to the service person, like their shoes, clothes, tie, jewellery, hairstyle, make-up, watch, cosmetics, perfume, etc." }

使用说明

• 加载数据集: python from datasets import load_dataset

  下载并加载所有配对

  dataset = load_dataset("ai4bharat/IN22-Gen", "all")

  下载并加载特定配对

  dataset = load_dataset("ai4bharat/IN22-Gen", "eng_Latn-hin_Deva")

涵盖语言

• 语言列表:
  • 阿萨姆语 (asm_Beng)
  • 孟加拉语 (ben_Beng)
  • 博多语 (brx_Deva)
  • 多格里语 (doi_Deva)
  • 英语 (eng_Latn)
  • 孔卡尼语 (gom_Deva)
  • 古吉拉特语 (guj_Gujr)
  • 印地语 (hin_Deva)
  • 卡纳达语 (kan_Knda)
  • 克什米尔语 (阿拉伯文) (kas_Arab)
  • 迈蒂利语 (mai_Deva)
  • 马拉雅拉姆语 (mal_Mlym)
  • 马拉地语 (mar_Deva)
  • 曼尼普尔语 (Bengali) (mni_Beng)
  • 曼尼普尔语 (Meitei) (mni_Mtei)
  • 尼泊尔语 (npi_Deva)
  • 奥里亚语 (ory_Orya)
  • 旁遮普语 (pan_Guru)
  • 梵语 (san_Deva)
  • 桑塔利语 (sat_Olck)
  • 信德语 (阿拉伯文) (snd_Arab)
  • 信德语 (Devanagari) (snd_Deva)
  • 泰米尔语 (tam_Taml)
  • 泰卢固语 (tel_Telu)
  • 乌尔都语 (urd_Arab)

引用

• 引用格式:

  @article{gala2023indictrans, title={IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages}, author={Jay Gala and Pranjal A Chitale and A K Raghavan and Varun Gumma and Sumanth Doddapaneni and Aswanth Kumar M and Janki Atul Nawale and Anupama Sujatha and Ratish Puduppully and Vivek Raghavan and Pratyush Kumar and Mitesh M Khapra and Raj Dabre and Anoop Kunchukuttan}, journal={Transactions on Machine Learning Research}, issn={2835-8856}, year={2023}, url={https://openreview.net/forum?id=vfT4YuzAYA}, note={} }