ajesujoba/twi_wordsim353

Yorùbá Wordsim-353数据集是wordsim-353数据集的Twi语言翻译版本，但只翻译了353对单词中的274对。该数据集包含单词对及其相似性评分，主要用于文本分类任务中的文本评分和语义相似性评分。数据集的语言为Twi（ISO 639-1: tw），数据集的创建过程涉及专家生成和众包注释。数据集的结构包括数据实例、数据字段和数据分割，其中数据字段包括twi1（第一个单词的Twi翻译）、twi2（第二个单词的Twi翻译）和similarity（相似性评分）。

原始信息汇总

数据集概述

数据集描述

数据集摘要

Yorùbá Wordsim-353 数据集是对 wordsim-353 词对相似度数据集的翻译，但仅翻译了 274 对（共 353 对）词。

支持的任务和排行榜

[更多信息需补充]

语言

Twi (ISO 639-1: tw)

数据集结构

数据实例

每个实例包含一对词及其相似度。数据集包含原始英语词（来自 wordsim-353）及其 Twi 翻译。

数据字段

• twi1: 词对中的第一个词；Twi 翻译
• twi2: 词对中的第二个词；Twi 翻译
• similarity: 根据英语数据集的相似度评分

数据分割

仅提供测试数据

数据集创建

策划理由

[更多信息需补充]

源数据

初始数据收集和规范化

[更多信息需补充]

源语言生产者

[更多信息需补充]

注释

注释过程

[更多信息需补充]

注释者

[更多信息需补充]

个人和敏感信息

[更多信息需补充]

使用数据的考虑因素

数据集的社会影响

[更多信息需补充]

偏见讨论

[更多信息需补充]

其他已知限制

[更多信息需补充]

附加信息

数据集策展人

[更多信息需补充]

许可信息

[更多信息需补充]

引用信息

@inproceedings{alabi-etal-2020-massive, title = "Massive vs. Curated Embeddings for Low-Resourced Languages: the Case of {Y}or{u}b{a} and {T}wi", author = "Alabi, Jesujoba and Amponsah-Kaakyire, Kwabena and Adelani, David and Espa{~n}a-Bonet, Cristina", booktitle = "Proceedings of the 12th Language Resources and Evaluation Conference", month = may, year = "2020", address = "Marseille, France", publisher = "European Language Resources Association", url = "https://www.aclweb.org/anthology/2020.lrec-1.335", pages = "2754--2762", abstract = "The success of several architectures to learn semantic representations from unannotated text and the availability of these kind of texts in online multilingual resources such as Wikipedia has facilitated the massive and automatic creation of resources for multiple languages. The evaluation of such resources is usually done for the high-resourced languages, where one has a smorgasbord of tasks and test sets to evaluate on. For low-resourced languages, the evaluation is more difficult and normally ignored, with the hope that the impressive capability of deep learning architectures to learn (multilingual) representations in the high-resourced setting holds in the low-resourced setting too. In this paper we focus on two African languages, Yor{u}b{a} and Twi, and compare the word embeddings obtained in this way, with word embeddings obtained from curated corpora and a language-dependent processing. We analyse the noise in the publicly available corpora, collect high quality and noisy data for the two languages and quantify the improvements that depend not only on the amount of data but on the quality too. We also use different architectures that learn word representations both from surface forms and characters to further exploit all the available information which showed to be important for these languages. For the evaluation, we manually translate the wordsim-353 word pairs dataset from English into Yor{u}b{a} and Twi. We extend the analysis to contextual word embeddings and evaluate multilingual BERT on a named entity recognition task. For this, we annotate with named entities the Global Voices corpus for Yor{u}b{a}. As output of the work, we provide corpora, embeddings and the test suits for both languages.", language = "English", ISBN = "979-10-95546-34-4", }

贡献

感谢 @dadelani 添加此数据集。