BIRCO Dataset

BIRCO is a collection of existing Information Retrieval datasets after carefull curation to make it suitable for Large Language Model (LLM) based systems evaluation. Here are the references for each of the 5 datasets used in BIRCO:1. DORIS-MAE: Wang, Jianyou Andre, et al. "Scientific document retrieval using multi-level aspect-based queries." Advances in Neural Information Processing Systems 36 (2024). (https://proceedings.neurips.cc/paper_files/paper/2023/hash/78f9c04bdcb06f1ada3902912d8b64ba-Abstract-Datasets_and_Benchmarks.html)2. ArguAna: Wachsmuth, Henning, Shahbaz Syed, and Benno Stein. "Retrieval of the best counterargument without prior topic knowledge." Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018. (https://aclanthology.org/P18-1023/)3. WhatThatBook: Lin, Kevin, et al. "Decomposing Complex Queries for Tip-of-the-tongue Retrieval." arXiv preprint arXiv:2305.15053 (2023). (https://arxiv.org/abs/2305.15053)4. Clinical-Trial: Koopman, Bevan, and Guido Zuccon. "A test collection for matching patients to clinical trials." Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. 2016. (https://dl.acm.org/doi/abs/10.1145/2911451.2914672)5. RELIC: Thai, Katherine, et al. "RELiC: Retrieving Evidence for Literary Claims." Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022. (https://aclanthology.org/2022.acl-long.517/)The dataset is stored as a json format. The structure of the file is as follows in python dict:├── ada_embedding_for_datasets_v1.pickle│├── "doris-mae"│   ├── "query" (60 queries)│   │   ├── query_id_1: "query text 1"│   │   ├── query_id_2: "query text 2"│   │   └── query_id_3: "query text 3"│   │   ...│   ├── "corpus" (5543 paper abstracts)│   │   ├── corpus_id_1: "corpus text 1"│   │   ├── corpus_id_2: "corpus text 2"│   │   └── corpus_id_3: "corpus text 3"│   │   ...│   └── "qrel" (avg. candidate pool size: 110.55)│       ├── query_id_1│       │   ├── corpus_id_1: relevance_score (rational number between 0-2)│       │   ├── corpus_id_2: relevance_score│       │   └── corpus_id_3: relevance_score│       │   ...│       ├── query_id_2│       │   ├── corpus_id_1: relevance_score│       │   ├── corpus_id_2: relevance_score│       │   └── corpus_id_3: relevance_score│       │   ...│       └── query_id_3│           ├── corpus_id_1: relevance_score│           ├── corpus_id_2: relevance_score│           └── corpus_id_3: relevance_score│           ...│├── "arguana" │   ├── "query" (100 queries)│   │   ├── query_id_1: "query text 1" │   │   ├── query_id_2: "query text 2"│   │   └── query_id_3: "query text 3"│   │   ...│   ├── "corpus" (3148 arguments)│   │   ├── corpus_id_1: "corpus text 1"│   │   ├── corpus_id_2: "corpus text 2"│   │   └── corpus_id_3: "corpus text 3"│   │   ...│   └── "qrel" (avg. candidate pool size: 50.01)│       ├── query_id_1│       │   ├── corpus_id_1: relevance_score (either 0 or 1)│       │   ├── corpus_id_2: relevance_score│       │   └── corpus_id_3: relevance_score│       │   ...│       ├── query_id_2│       │   ├── corpus_id_1: relevance_score│       │   ├── corpus_id_2: relevance_score│       │   └── corpus_id_3: relevance_score│       │   ...│       └── query_id_3│           ├── corpus_id_1: relevance_score│           ├── corpus_id_2: relevance_score│           └── corpus_id_3: relevance_score│           ...│├── "wtb" │   ├── "query" (100 queries)│   │   ├── query_id_1: "query text 1"│   │   ├── query_id_2: "query text 2"│   │   └── query_id_3: "query text 3"│   │   ...│   ├── "corpus" (1767 book descriptions)│   │   ├── corpus_id_1: "corpus text 1"│   │   ├── corpus_id_2: "corpus text 2"│   │   └── corpus_id_3: "corpus text 3"│   │   ...│   └── "qrel" (avg. candidate pool size: 50.43)│       ├── query_id_1│       │   ├── corpus_id_1: relevance_score (either 0 or 1)│       │   ├── corpus_id_2: relevance_score│       │   └── corpus_id_3: relevance_score│       │   ...│       ├── query_id_2│       │   ├── corpus_id_1: relevance_score│       │   ├── corpus_id_2: relevance_score│       │   └── corpus_id_3: relevance_score│       │   ...│       └── query_id_3│           ├── corpus_id_1: relevance_score│           ├── corpus_id_2: relevance_score│           └── corpus_id_3: relevance_score│           ...│├── "clinical-trial" (avg. candidate pool size )│   ├── "query" (50 queries)│   │   ├── query_id_1: "query text 1"│   │   ├── query_id_2: "query text 2"│   │   └── query_id_3: "query text 3"│   │   ...│   ├── "corpus" (3256 clinical trial descriptions)│   │   ├── corpus_id_1: "corpus text 1"│   │   ├── corpus_id_2: "corpus text 2"│   │   └── corpus_id_3: "corpus text 3"│   │   ...│   └── "qrel" (avg. candidate pool size: 68.40)│       ├── query_id_1│       │   ├── corpus_id_1: relevance_score (0, 1, or 2)│       │   ├── corpus_id_2: relevance_score│       │   └── corpus_id_3: relevance_score│       │   ...│       ├── query_id_2│       │   ├── corpus_id_1: relevance_score│       │   ├── corpus_id_2: relevance_score│       │   └── corpus_id_3: relevance_score│       │   ...│       └── query_id_3│           ├── corpus_id_1: relevance_score│           ├── corpus_id_2: relevance_score│           └── corpus_id_3: relevance_score│           ...│└── "relic"     ├── "query" (100 queries)    │   ├── query_id_1: "query text 1"    │   ├── query_id_2: "query text 2"    │   └── query_id_3: "query text 3"    │   ...    ├── "corpus" (5017 quotations from books)    │   ├── corpus_id_1: "corpus text 1"    │   ├── corpus_id_2: "corpus text 2"    │   └── corpus_id_3: "corpus text 3"    │   ...    └── "qrel" (avg. candidate pool size: 50.59)        ├── query_id_1        │   ├── corpus_id_1: relevance_score (either 0 or 1)        │   ├── corpus_id_2: relevance_score        │   └── corpus_id_3: relevance_score        │   ...        ├── query_id_2        │   ├── corpus_id_1: relevance_score        │   ├── corpus_id_2: relevance_score        │   └── corpus_id_3: relevance_score        │   ...        └── query_id_3            ├── corpus_id_1: relevance_score            ├── corpus_id_2: relevance_score            └── corpus_id_3: relevance_score            ...