2nd International Workshop on Open Web Search #wows2025 at ECIR 2025: WOWS-EVAL

This repository hosts the data for the WOWS-EVAL shared task where we aim to collaboratively and cooperatively enrich the Open Web Index (OWI) with relevance judgments transferred from existing TREC-like test collections. For this, we develop and collect automated relevance assessors that use a TREC-style information need and a document as input to output a probability that the document is relevant to the query. The test set consists of information needs from existing TREC test collections for which we will create a new round of relevance judgments to evaluate the effectiveness of submitted relevance assessors. We encourage run submissions with ir-metadata and we aim to dockerize the most effective submissions after the shared task to apply them to the Open Web Index to allow first effectiveness evaluations on the Open Web Index.