Rank-DistiLLM Novelty

This dataset contains the novelty ranking data for the paper Set-Encoder: Permutation-Invariant Inter-Passage Attention for Listwise Passage Re-Ranking with Cross-Encoders. The dataset is based off the Rank-Distillm dataset (Paper, Dataset). The two files contain the top 100 retrieved passages for 10k queries from the MS MARCO passage training dataset re-ranked by the RankZephyr model for BM25 and ColBERTv2, respectively. The central difference to the Rank-DistiLLM dataset files is that the run files in this dataset are additionally grouped by lexical similarity. All passages within a ranking a clustered according to their jaccard similarity. Any passages with a jaccard similarity > 0.5 are grouped into a cluster and have the same id in the second column (the Q0 column in the standard TREC run format).