SemEval-2024 Task 9: BRAINTEASER: A Novel Task Defying Common Sense

Data  for the SemEval 2024 paper "SemEval-2024 Task 9: BRAINTEASER: A Novel Task Defying Common Sense"

The data of the two subtasks is saved in the data folder, BTDATA.zip, which contains the data for the sentence puzzle and word puzzle.

The data contained in BTDATA.zip are as follows:

Semeval Competition

Training Data

SP_train.npy (Semeval training data)

WP_train.npy (Semeval training data)

Test Data

SP_test.npy (Semeval test data)

WP_test.npy (Semeval test data)

SP_test_answer.npy (Semeval test data answer)

WP_test_answer.npy (Semeval test data answer)

Relation to EMNLP 2023 Paper

The relationship between EMNLP and SemEval involves using the same dataset but with different data splitting and utilization methods. In EMNLP, the entire dataset is employed for testing, while in SemEval, the dataset is divided into training and testing sets, with the training set comprising a significant majority.

Our EMNLP paper results on GitHub are tested on the entire data in a zero-shot manner. In the SemEval2024-Task9, although the whole dataset is the same as our EMNLP paper, we allow people to train on 80% of the whole dataset, and we evaluate the system on the 20% left.

EMNLP Zero-Shot Experiment

sentence_puzzle.npy (on all sentence puzzle data)

word_puzzle.npy (on all word puzzle data)

Note: To prevent automatic data crawlers, BTDATA.zip needs a password: brainteaser