ir_metadata: An Extensible Metadata Schema for Information Retrieval Experiments

This dataset accompanies our work that introduces a metadata schema for TREC run files based on the PRIMAD model. PRIMAD considers essential components of computational experiments that possibly can affect reproducibility on a conceptual level. We propose to align the metadata annotations to the PRIMAD components. In order to demonstrate the potential of metadata annotations, we curated a dataset with run files derived from experiments with different instantiations of PRIMAD components and annotated these with the corresponding metadata. With this work, we hope to stimulate IR researchers to annotate run files and improve the reuse value of experimental artifacts even further. This archive contains the following data: <strong>demo.tar.xz</strong> : Selected annotated runs files that are used in the Colab demonstration. <strong>metadata.zip</strong> : YAML files containing only the metadata annotations for each run. <strong>runs.zip</strong> : The entire set of run files with annotations. The annotated runs result from the following experiments: Grossman and Cormack @ TREC Common Core 2017 Paper | Source Grossman and Cormack @ TREC Common Core 2018 Paper | Source Yu et al. @ TREC Common Core 2018 Paper | Source Yu et al. @ ECIR 2019 Paper | Source Breuer et al. @ SIGIR 2020 Paper | Source Breuer et al. @ CLEF 2021 Paper | Source

本数据集配套于一项基于PRIMAD模型（PRIMAD）构建TREC检索结果文件（TREC run files）元数据架构的研究工作。PRIMAD模型从概念层面梳理了可能影响实验可复现性的计算实验核心构成要素。本研究提出将元数据标注与PRIMAD模型的核心构成要素进行对齐。为验证元数据标注的应用潜力，我们整理了一套数据集：该数据集包含源自采用不同PRIMAD要素实例化方案的实验所生成的检索结果文件，并为这些文件标注了对应元数据。藉由本项工作，我们期望推动信息检索（Information Retrieval, IR）领域研究者开展检索结果文件的标注工作，进一步提升实验成果的复用价值。本归档文件包含以下数据：
**demo.tar.xz**：用于Colab演示的精选标注检索结果文件
**metadata.zip**：仅包含各检索结果文件元数据标注的YAML格式文件
**runs.zip**：完整带标注的检索结果文件集合
本次标注的检索结果文件源自以下实验：
Grossman与Cormack发表于TREC Common Core 2017的论文 | 来源
Grossman与Cormack发表于TREC Common Core 2018的论文 | 来源
Yu等人发表于TREC Common Core 2018的论文 | 来源
Yu等人发表于ECIR 2019的论文 | 来源
Breuer等人发表于SIGIR 2020的论文 | 来源
Breuer等人发表于CLEF 2021的论文 | 来源