SemEval-2025 Task 7: Multilingual and Crosslingual Fact-Checked Claim Retrieval

SemEval-2025 Task 7

We provide here the dataset used in the SemEval-2025 shared task 7 on Multilingual and Crosslingual Fact-Checked Claim Retrieval, which addressed the critical challenge of efficiently identifying previously fact-checked claims across multiple languages — a task that can be time-consuming for professional fact-checkers even within a single language and becomes much more difficult to perform manually when the claim and the fact-check may be in different languages. Given the global spread of disinformation narratives, the range of languages that one would need to cover not to miss existing fact-checks is vast.

More specifically, the shared task was formulated as follows: Given social media posts, the goal is to retrieve the most relevant claims from a list of claims that were previously fact-checked.

Dataset description

The dataset consists of the following files:

train_dev_set containing training and development set data files

test_set containing test set data files

scripts to evaluate the monolingual and crosslingual retrieval as well as to load the dataset

baselines containing test set predictions for 4 models, namely: BM25, GTR-T5-Large, Paraphrase-Multilingual-MPNet-Base-v2, and Multilingual-E5

README.md containing more information on the data format and how to use the provided scripts

Relation to the MultiClaim dataset

The training and development sets contained in the dataset are based on the MultiClaim dataset published with our EMNLP 2023 paper. The test set contains additional data, which are not part of the original MultiClaim dataset, but which were collected using the same methodology. For more details, see the shared task paper.

Since the original MultiClaim dataset, part of which is included in this dataset, was published for research purposes only, we publish this dataset under the same terms and provide access to it upon request. 

References

If you use this dataset in any publication, project, tool or in any other form, please, cite the following papers:

@inproceedings{semeval2025task7,    title=”{S}em{E}val-2025 Task 7: Multilingual and Crosslingual Fact-Checked Claim Retrieval”,    author=”Peng, Qiwei and Moro, Robert and Gregor, Michal and Srba, Ivan and Ostermann, Simon and Simko, Marian and Podroužek, Juraj and Mesarčík, Matúš and Kopčan, Jaroslav and Søgaard, Anders”,    booktitle = “Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025)”,    year = {2025},    address = {Vienna, Austria},    month = {July},    publisher = "Association for Computational Linguistics",}

@inproceedings{pikuliak-etal-2023-multilingual,
title = "Multilingual Previously Fact-Checked Claim Retrieval",
author = "Pikuliak, Mat{\'u}{\v{s}} and Srba, Ivan and Moro, Robert and Hromadka, Timo and Smole{\v{n}}, Timotej and Meli{\v{s}}ek, Martin and Vykopal, Ivan and Simko, Jakub and Podrou{\v{z}}ek, Juraj and Bielikova, Maria",
editor = "Bouamor, Houda and Pino, Juan and Bali, Kalika",
booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",
month = dec,
year = "2023",
address = "Singapore",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.emnlp-main.1027",
doi = "10.18653/v1/2023.emnlp-main.1027",
pages = "16477--16500",
}