rank1-training-data

# rank1-training-data: Training Dataset for rank1 Reasoning Rerankers 📄 [Paper](https://arxiv.org/abs/2502.18418) | 🚀 [GitHub Repository](https://github.com/orionw/rank1) This dataset contains the training data used to develop the rank1 family of reasoning rerankers with [LLaMA Factory](https://github.com/hiyouga/LLaMA-Factory/tree/main). It includes query-document pairs with relevance judgments and reasoning chains that guided the models to make binary relevance decisions. ## Dataset Description The rank1-training-data dataset is a comprehensive collection of training examples used to teach rank1 models how to reason about document relevance. Each example contains: 1. A query 2. A document/passage 3. A reasoning chain that analyzes the relevance in a step-by-step manner 4. A binary relevance judgment (true/false) The dataset is designed to train models to generate explicit reasoning chains before making relevance judgments, which helps break down complex relevance decisions into logical steps. ## Dataset Structure The dataset follows this structure: ``` { "instruction": "The general task instruction", "input": "The search query text and passage text", "output": "The reasoning chain to generate and have loss on", } ``` ## Associated Models and Resources | Resource | Description | |:---------|:------------| | [rank1-7b](https://huggingface.co/jhu-clsp/rank1-7b) | Base rank1 model (7B parameters) | | [rank1-14b](https://huggingface.co/jhu-clsp/rank1-14b) | Larger rank1 variant (14B parameters) | | [rank1-32b](https://huggingface.co/jhu-clsp/rank1-32b) | Largest rank1 variant (32B parameters) | | [rank1-mistral-2501-24b](https://huggingface.co/jhu-clsp/rank1-mistral-2501-24b) | Mistral-based rank1 variant (24B parameters) | | [rank1-llama3-8b](https://huggingface.co/jhu-clsp/rank1-llama3-8b) | Llama 3.1-based rank1 variant (8B parameters) | | [rank1-r1-msmarco](https://huggingface.co/datasets/jhu-clsp/rank1-r1-msmarco) | All R1 output examples from MS MARCO | | [rank1-run-files](https://huggingface.co/datasets/jhu-clsp/rank1-run-files) | Pre-computed run files for top-100 doc reranking | ## Usage Examples See the Github repo for the training configurations. ## Citation If you use this dataset in your research, please cite: ```bibtex @misc{weller2025rank1testtimecomputereranking, title={Rank1: Test-Time Compute for Reranking in Information Retrieval}, author={Orion Weller and Kathryn Ricci and Eugene Yang and Andrew Yates and Dawn Lawrie and Benjamin Van Durme}, year={2025}, eprint={2502.18418}, archivePrefix={arXiv}, primaryClass={cs.IR}, url={https://arxiv.org/abs/2502.18418}, } ``` ## License [MIT License](https://github.com/orionw/rank1/blob/main/LICENSE)