SemEval-2024 Task 9: BRAINTEASER: A Novel Task Defying Common Sense
收藏资源简介：
Data for the SemEval 2024 paper "SemEval-2024 Task 9: BRAINTEASER: A Novel Task Defying Common Sense" The data of the two subtasks is saved in the data folder, BTDATA.zip, which contains the data for the sentence puzzle and word puzzle. The data contained in BTDATA.zip are as follows: Semeval Competition Training Data SP_train.npy (Semeval training data) WP_train.npy (Semeval training data) Test Data SP_test.npy (Semeval test data) WP_test.npy (Semeval test data) SP_test_answer.npy (Semeval test data answer) WP_test_answer.npy (Semeval test data answer) Relation to EMNLP 2023 Paper The relationship between EMNLP and SemEval involves using the same dataset but with different data splitting and utilization methods. In EMNLP, the entire dataset is employed for testing, while in SemEval, the dataset is divided into training and testing sets, with the training set comprising a significant majority. Our EMNLP paper results on GitHub are tested on the entire data in a zero-shot manner. In the SemEval2024-Task9, although the whole dataset is the same as our EMNLP paper, we allow people to train on 80% of the whole dataset, and we evaluate the system on the 20% left. EMNLP Zero-Shot Experiment sentence_puzzle.npy (on all sentence puzzle data) word_puzzle.npy (on all word puzzle data) Note: To prevent automatic data crawlers, BTDATA.zip needs a password: brainteaser
SemEval 2024 论文《SemEval-2024 任务9：BRAINTEASER：一项挑战常识的新颖任务》所用数据集 本任务两个子任务的数据均存储于data文件夹下的BTDATA.zip压缩包中，该压缩包包含句子谜题与单词谜题两类数据集。 BTDATA.zip内包含的数据如下： SemEval竞赛数据集 训练数据： SP_train.npy（Semeval训练数据） WP_train.npy（Semeval训练数据） 测试数据： SP_test.npy（Semeval测试数据） WP_test.npy（Semeval测试数据） SP_test_answer.npy（Semeval测试数据答案） WP_test_answer.npy（Semeval测试数据答案） 与EMNLP 2023论文的关联 本次数据集与EMNLP 2023论文所用数据集完全一致，但二者的数据划分与使用方式存在差异。在EMNLP的研究中，全体数据集均被用于测试；而在本次SemEval任务中，数据集被划分为训练集与测试集，其中训练集占绝大多数。 我们在GitHub上发布的EMNLP论文实验结果，是基于全量数据集以零样本（zero-shot）方式完成测试的。尽管本次SemEval 2024 任务9所使用的全量数据集与EMNLP论文一致，但我们允许参赛者在全量数据的80%上进行模型训练，并在剩余20%的数据上对系统进行评估。 EMNLP零样本实验所用数据： sentence_puzzle.npy（对应全部句子谜题数据） word_puzzle.npy（对应全部单词谜题数据） 注意：为防止自动爬虫程序爬取，BTDATA.zip的解压密码为：brainteaser
