Early Irish Analogy Dataset for Word Embedding Evaluation

An embedding evaluation dataset for Early Irish described in the paper "Do not Trust the Experts: How the Lack of Standard Complicates NLP for Historical Irish".

Traditionally, analogy datasets are based on pairwise semantic proportion, and therefore every question has a single correct answer. Given the high level of variation in historical languages, such a strict definition of a correct answer seems unjustified. Therefore, Early Irish Analogy Dataset follows the Bigger Analogy Test Set (BATS) and provides several correct answers to each analogy question. 

Morphological and spelling variation data are extracted from the eDIL, a historical dictionary of medieval Irish. Unlike BATS, no distinction is made between inflection types due to eDIL's structure. The raw data amounted to 2,370 spelling variation and 9,690 morphological variation questions, from which 150 examples were randomly selected for each of the subsets to be comparable in size with the synonym and antonym subsets. The synonym and antonym subsets are translations of the correspondent BATS parts obtained by reverse-searching the eDIL and proofread by four expert evaluators. The dataset includes 98 entries in the synonym subset and 109 entries in the antonym subset, upon which three or more experts agreed.

本数据集为早期爱尔兰语（Early Irish）嵌入模型评估数据集，相关研究成果刊载于论文《切勿轻信专家：标准缺失如何为古爱尔兰语自然语言处理（NLP）带来挑战》（"Do not Trust the Experts: How the Lack of Standard Complicates NLP for Historical Irish"）。
传统上，类比数据集均基于成对语义比例构建，因此每个问题仅存在唯一正确答案。鉴于历史语言存在高度变异，这种对正确答案的严格定义并不合理。因此，早期爱尔兰语类比数据集（Early Irish Analogy Dataset）参考大类比测试集（Bigger Analogy Test Set, BATS）的范式，为每个类比问题提供多个正确答案。
形态变异与拼写变异数据均提取自中世纪爱尔兰语历史词典eDIL。与BATS不同，受eDIL的数据库结构限制，本数据集未对屈折类型进行区分。原始数据集包含2370条拼写变异问题与9690条形态变异问题，随后从这两类子集各随机选取150条样本，以确保其规模与同义词、反义词子集相当。同义词与反义词子集为对应BATS中相应部分的翻译结果，通过反向检索eDIL得到，并经四位专家评估校对。
本数据集的同义词子集包含98条经三位及以上专家一致认可的条目，反义词子集则包含109条同类认可条目。