2nd International Workshop on Open Web Search #wows2025 at ECIR 2025: WOWS-EVAL

This repository hosts the data for the WOWS-EVAL shared task where we aim to collaboratively and cooperatively enrich the Open Web Index (OWI) with relevance judgments transferred from existing TREC-like test collections. For this, we develop and collect automated relevance assessors that use a TREC-style information need and a document as input to output a probability that the document is relevant to the query. The test set consists of information needs from existing TREC test collections for which we will create a new round of relevance judgments to evaluate the effectiveness of submitted relevance assessors. We encourage run submissions with ir-metadata and we aim to dockerize the most effective submissions after the shared task to apply them to the Open Web Index to allow first effectiveness evaluations on the Open Web Index.

本仓库存储了WOWS-EVAL共享任务的配套数据集。该共享任务旨在通过从现有类TREC测试集迁移相关性标注的方式，协作式地丰富开放网页索引（Open Web Index，OWI）。为此，我们开发并收集了自动化相关性评估器，此类评估器以类TREC格式的信息需求与文档作为输入，输出该文档与对应查询相关的概率值。本次任务的测试集取自现有TREC测试集中的信息需求，我们将为这些信息需求生成新一轮的相关性标注，以此评估提交的自动化相关性评估器的检索效果。我们鼓励提交附带信息检索元数据（ir-metadata）的运行结果，并计划在共享任务结束后，将表现最优的提交结果进行Docker容器化处理，以便将其应用于开放网页索引，从而实现针对该索引的首次检索效果评估。