XGLUE

"XGLUE 是一个新的基准数据集，用于评估跨语言预训练模型在跨语言自然语言理解和生成方面的性能。对于从事 XTREME 工作的人，我们建议您将结果提交给 XGLUE同样，特别是对于 XGLUE 独特的任务，因为 (1) XGLUE 包括来自现实世界场景的任务（新闻分类、查询-广告匹配、网页排名和 QA 匹配），所有这些数据集都是由人工生成和标记的， (2) XGLUE 提供了两个跨语言文本生成任务。XGLUE 由 11 个任务组成，跨越 19 种语言。对于每个任务，训练数据只有英文。这意味着要在 XGLUE 上取得成功，一个模型必须具有很强的零样本跨语言迁移能力，才能从特定任务的英文数据中学习，并将所学的东西迁移到其他语言中。相比于它的并发工作 XTREME，XGLUE 有两个特点：第一，它包括跨语言李ngual NLU 和跨语言 NLG 任务同时进行；其次，除了包含 5 个现有的跨语言任务（即 NER、POS、MLQA、PAWS-X 和 XNLI）外，XGLUE 还从 Bing 场景中选择了 6 个新任务，包括新闻分类（NC）、查询广告匹配（QADSM） 、网页排名 (WPR)、QA 匹配 (QAM)、问题生成 (QG) 和新闻标题生成 (NTG)。这种语言、任务和任务来源的多样性为量化跨语言自然语言理解和生成的预训练模型的质量提供了一个全面的基准。”

XGLUE is a novel benchmark dataset for evaluating the performance of cross-lingual pre-trained models on cross-lingual natural language understanding (NLU) and generation (NLG). For researchers working on XTREME, we recommend submitting results to XGLUE as well, especially for tasks unique to XGLUE, due to the following two reasons: (1) XGLUE includes tasks derived from real-world scenarios, namely news classification, query-advertisement matching, web ranking, and QA matching, all of which are manually generated and labeled; (2) XGLUE provides two cross-lingual text generation tasks. XGLUE consists of 11 tasks spanning 19 languages. For each task, the training data is only available in English. This implies that to achieve strong performance on XGLUE, a model must possess robust zero-shot cross-lingual transfer capability: it needs to learn from task-specific English data and transfer the acquired knowledge to other languages. Compared with its concurrent work XTREME, XGLUE has two distinct features: First, it covers both cross-lingual NLU and cross-lingual NLG tasks simultaneously; Second, in addition to including 5 existing cross-lingual tasks (i.e., NER, POS, MLQA, PAWS-X and XNLI), XGLUE also selects 6 new tasks from Bing scenarios, including News Classification (NC), Query-Advertisement Matching (QADSM), Web Page Ranking (WPR), QA Matching (QAM), Question Generation (QG) and News Title Generation (NTG). This diversity in languages, tasks and task sources provides a comprehensive benchmark for quantifying the quality of pre-trained models for cross-lingual natural language understanding and generation.

搜集汇总

背景与挑战

背景概述

XGLUE是一个用于评估跨语言预训练模型性能的基准数据集，涵盖11个任务和19种语言，专注于自然语言理解和生成。该数据集仅提供英文训练数据，旨在测试模型的零样本跨语言迁移能力，任务源自现实场景如新闻分类和查询广告匹配，为跨语言模型提供全面评估。