IR Lab Cologne/Jena/Kassel Winter Term 2024/2025

The Datasets for the Information Retrieval Courses in Cologne/Jena/Kassel in Winter Term 2024/2025 This repository contains resources coupled to ir_datasets and TIREx for IR courses that focus their hands-on labs on shared tasks. During the IR exercises in winter term 2023/2024, we collaboratively developed and evaluated IR systems in a shared task style setup, covering corpus creation, system development, and statistical analysis. The resulting artifacts, i.e., the documents, topics, runs, relevance judgments can be browsed at https://tira.io/task-overview/ir-lab-wise-2024. This zenodo artifact contains all of the underlying datasets used and produced during the course together with instructions on how to easily access the data using ir_datasets. The artifact in this dataset include the following files: subsampled-ms-marco-deep-learning-20241201-training-inputs.zip containing the training inputs, i.e., containing the document corpus and the topics. subsampled-ms-marco-deep-learning-20241201-training-truths.zip containing the training truth to evaluate and tune systems, i.e., the topics and relevance judgments. Accessing the Data with ir_datasets We provide wrapper code to easily access the resources with ir_datasets: # this loads a patched version of ir_datasets that can load resources from TIRA from tira.third_party_integrations import ir_datasets training_dataset = ir_datasets.load('ir-lab-wise-2024/subsampled-ms-marco-deep-learning-20241201-training') Similarly, the same is possible with the ir_datasets integration to PyTerrier: from tira.third_party_integrations import ensure_pyterrier_is_loaded import pyterrier as pt # this patches ir_datasets and loads PyTerrier so that it can load resources from TIRA and can run in the TIRA sandbox ensure_pyterrier_is_loaded() training_dataset = pt.datasets.get_dataset('irds:ir-lab-wise-2024/subsampled-ms-marco-deep-learning-20241201-training')