Now that your System has been Reproduced, What does this Mean for the Users?

This data repository corresponds to our reproducibility experiment entitled "Now that your System has been Reproduced, What does this Mean for the Users?". It contains the ranking results that were used in the online experiments (rankings.json). The corresponding source code to generate the rankings can be found in the zipped src folder and also on the GitHub repository. The retrieval pipeline is based on a two-stage approach with BM25 [1] as a lexical first-stage retriever and monoT5 [2] as a second-stage reranker. For each stage, this repository holds a separate zipped directory with: 

- plain files: rankings in TREC-formatted style,- compressed files: the above in a compressed format (tar.gz),- jsonl files: the top 10 results for each query in JSON format.

The final rankings used for the online experiments with additional metadata, including the retrieval effectiveness and reproducibility scores, as well as the generated passage title are contained in the rankings.json file. Please take a look at the GitHub repository regarding further information of how to run the scripts in order to reproduce the data artifacts.

[1] Robertson et al. 1994. Okapi at TREC-3. NIST. http://trec.nist.gov/pubs/trec3/papers/city.ps.gz[2] Pradeep et al. 2021. The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models. arXiv. https://arxiv.org/abs/2101.05667