zetavg/mlqa_en_zh_tw
收藏资源简介：
MLQA en-zh_tw数据集是MLQA（多语言问答）数据集的中英双语版本，特别转换为台湾正体中文，并合并了中文和英文版本的相同项目，以便于双语语言模型使用。数据集分为dev和test两个部分，分别包含302和2986组数据。转换过程包括使用OpenCC将简体中文转换为台湾正体中文，使用pangu.js在中英文之间添加空格，以及合并中英文数据集中的相同项目。已知问题包括某些项目可能缺少一种语言的版本，部分问题与答案可能存在理解偏误或歧异，以及不同语言版本的context长度和内容范围可能有很大差异。
The MLQA en-zh_tw dataset is a Chinese-English bilingual variant of the MLQA (Multilingual Question Answering) dataset, where the Chinese portion has been specially converted to Taiwan Traditional Chinese, and identical entries from the original Chinese and English versions of the dataset are merged to facilitate use by bilingual language models. The dataset is divided into two subsets: dev and test, containing 302 and 2986 data samples respectively. The conversion procedures include using OpenCC to convert Simplified Chinese to Taiwan Traditional Chinese, using pangu.js to insert spaces between Chinese and English text, and merging identical entries across the Chinese and English datasets. Known issues include that some entries may be missing a version in one of the two languages, some questions and answers may contain comprehension biases or ambiguities, and the context length and content scope of different language versions may differ considerably.
数据集概述
基本信息
- 许可证：CC-BY-3.0
- 任务类别：
- 问答
- 翻译
- 语言：
- 中文
- 英文
- 数据集大小：1K<n<10K
- 美观名称：MLQA en-zh_tw
数据集内容
- 描述：MLQA (MultiLingual Question Answering) 中英双语问答资料集，为原始 MLQA 资料集转换为台湾正体中文的版本，并合并中文与英语版本的相同项目，方便供双语语言模型使用。
- 数据划分：分为
dev和test两个 split，分别包含 302 和 2986 组数据。
数据示例
json [ { "title": { "en": "Curling at the 2014 Winter Olympics", "zh_tw": "2014 年冬季奧林匹克運動會冰壺比賽" }, "paragraphs": [ { "context": { "en": "Qualification to the curling tournaments at the Winter Olympics was determined through two methods. Nations could qualify teams by earning qualification points from performances at the 2012 and 2013 World Curling Championships. Teams could also qualify through an Olympic qualification event which was held in the autumn of 2013. Seven nations qualified teams via World Championship qualification points, while two nations qualified through the qualification event. As host nation, Russia qualified teams automatically, thus making a total of ten teams per gender in the curling tournaments.", "zh_tw": "本屆冬奧會冰壺比賽參加資格有兩種辦法可以取得。各國家或地區可以透過 2012 年和 2013 年的世界冰壺錦標賽，也可以透過 2013 年 12 月舉辦的一次冬奧會資格賽來取得資格。七個國家透過兩屆世錦賽積分之和來獲得資格，兩個國家則透過冬奧會資格賽。作為主辦國，俄羅斯自動獲得參賽資格，這樣就確定了冬奧會冰壺比賽的男女各十支參賽隊伍。" }, "qas": [ { "id": "b08184972e38a79c47d01614aa08505bb3c9b680", "question": { "zh_tw": "俄羅斯有多少隊獲得參賽資格？", "en": "How many teams did Russia qualify for?" }, "answers": { "en": [ { "text": "ten teams", "answer_start": 543 } ], "zh_tw": [ { "text": "十支", "answer_start": 161 } ] } } ] } ] } ]
已知问题
- 有些项目的
title、paragraph的context、问题或是答案可能缺少其中一种语言的版本。 - 部分问题与答案可能存在理解偏误或歧义。
paragraph的context在不同语言的版本下可能长度与涵盖的内容范围有很大落差。
