TRECCOVID-RF-Aspire

This is a copy of the TRECCOVID-RF dataset used in the paper "Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similarity" by Sheshera Mysore, Arman Cohan, Tom Hope. The TRECCOVID dataset presents an ad-hoc search dataset. The versions of the original dataset used may be accessed here: query topics, relevance annotations, and the metadata for papers is obtained from the CORD-19 dataset in the 2021-06-21 release. The dataset released here converts the original TRECCOVID dataset into a reformulated form, TRECCOVID-RF which is used in the paper. <br><br>See further details of the paper, how this dataset was compiled, and how it was used: https://github.com/allenai/aspire<br>The contents of the dataset are as follows: <br><code>abstracts-treccovid.jsonl</code>: <code>jsonl</code> file containing the paper-id, abstracts, and titles for the queries and candidates which are part of the dataset.<br> <code><code>treccovid</code>-queries-release.csv</code>: Metadata associated with every query.<code>test-pid2anns-</code><code><code>treccovid</code>.json</code>: JSON file with the query paper-id, candidate paper-ids for every query paper in the dataset. Use these files in conjunction with <code>abstracts-</code><code><code>treccovid</code>.jsonl</code> to generate files for use in model evaluation. <br><code><code>treccovid</code>-evaluation_splits.json</code>: Paper-ids for the splits to use in reporting evaluation numbers. <code>aspire/src/evaluation/ranking_eval.py</code> included in the github repo accompanying this dataset implements the evaluation protocol and computes evaluation metrics. Please see the paper for descriptions of the experimental protocol we recommend to report evaluation metrics.<br>