MedHopQA: A Disease-Centered Multi-Hop Reasoning Benchmark for LLM-Based Biomedical Question Answering

MedHopQA is an English-language, disease-centered benchmark for evaluating multi-hop reasoning in biomedical question answering. The core benchmark comprises 1,000 expert-curated, open-ended question-answer pairs introduced as BioCreative IX Track 1. Each question is designed to require synthesis of information across two distinct, interconnected English Wikipedia articles; no single source is intended to be sufficient. The questions focus on diseases, genes and proteins, chemicals and medications, signs and symptoms, treatments, anatomy, and related biomedical concepts, with particular emphasis on rare diseases. The resource was constructed through a multi-stage human-AI workflow combining source-page selection, human annotation, AI-assisted augmentation, blind triage, iterative verification, and final answer validation. Canonical answers were augmented with human-reviewed lexical and concept-level variants drawn from resources including MONDO, NCBI Gene, NCBI Taxonomy, Wikipedia, and other supporting sources to accommodate semantically equivalent biomedical answers. For blind public evaluation, the 1,000 scored questions are embedded within a downloadable package of approximately 10,000 questions. The scored-item mapping and gold test answers are withheld to reduce leaderboard gaming and benchmark contamination. CodaBench is the live submission and leaderboard environment. This Zenodo record preserves the public data package and documentation for reproducible use and citation; it does not disclose the hidden test answer key or the mapping that identifies the scored subset. MedHopQA is intended for research on biomedical question answering, multi-step reasoning, retrieval-augmented generation, and evaluation of large language models. The benchmark is derived from English Wikipedia and is disease-centered, so it may reflect source coverage, source bias, and time-sensitive information. It is not a clinical decision-support resource and should not be used as a substitute for professional medical advice.