QADO: An RDF Representation of Question Answering Datasets and their Analyses for Improving Reproducibility

Measuring the quality of Question Answering (QA) systems is a crucial task to validate the results of novel approaches. However, there are already indicators of a reproducibility crisis as many published systems have used outdated datasets or use subsets of QA benchmarks, making it hard to compare results. We identified the following core problems: there is no standard data format, instead, proprietary data representations are used by the different datasets; additionally, we recognized a lack of understanding of which test questions are reused across different instances of a benchmark series; last but not least, the characteristics of datasets are typically not reflected by the dataset maintainers nor by the system publishers. To overcome these problems, we established an ontology---Question Answering Dataset Ontology (QADO)---for representing the QA datasets in RDF. The following datasets were mapped into the ontology: the QALD series, LC-QuAD series, RuBQ series, CWQ, and Mintaka. Hence, the integrated data in QADO covers widely used datasets and multilinguality. Additionally, we did intensive analyses of the datasets to identify their characteristics. The provided resource will enable the research community to improve the quality of their research and support the reproducibility of experiments. <br> Here, the mapping results of the QADO process, and the SPARQL queries for data analytics are provided.