ReMedQA

# ReMedQA This repository hosts the **ReMedQA** dataset, introduced in our paper: "[**ReMedQA: Are We Done With Medical Multiple-Choice Benchmarks?**](https://disi-unibo-nlp.github.io/remedqa/)" ## Overview While medical multiple-choice question answering (MCQA) benchmarks often report near-human accuracy, raw accuracy alone does not reliably measure a model’s true competence. Models may change answers under minor perturbations, exposing fragility and lack of robustness. **ReMedQA** addresses this limitation by extending standard medical MCQA datasets—**MedQA**, **MedMCQA**, and **MMLU-med**—with: * **Open-answer variants** (Open setting) * **MCQA systematically perturbed items** (Closed setting) Building on this design, we introduce *ReAcc* and *ReCon*, two reliability-oriented metrics. *ReAcc* captures the proportion of questions answered correctly across all variations (i.e., open and closed settings), whereas *ReCon* measures answer consistency across variations regardless of correctness. By jointly tracking accuracy and consistency, our metrics elevate reliability to a core dimension of evaluation—broadly applicable, yet critical in the medical domain. --- ## 📊 Dataset Structure The dataset is organized into the following medical subsets, along with their sizes and available pertubation modes: | Subset | # Samples | Available Modes | | --------- | --------- | ---------------------------------------------------------------------------------------------- | | `medqa` | 1,259 | `mcq`, `open`, `incorrect`, `roman_numeral`, `none_of_the_provided`, `fixed_pos`, `no_symbols` | | `medmcqa` | 1,000 | `mcq`, `open`, `incorrect`, `roman_numeral`, `none_of_the_provided`, `fixed_pos`, `no_symbols` | | `mmlu` | 895 | `mcq`, `open`, `incorrect`, `roman_numeral`, `none_of_the_provided`, `fixed_pos`, `no_symbols` | **Note:** For `mmlu`, only the following medical subjects were included: `clinical knowledge`, `medical genetics`, `anatomy`, `professional medicine`, `college biology`, and `college medicine`. --- # Available Perturbation Modes | Mode | Description | | ------------------------------ | --------------------------------------------------------------------------------------- | | `mcq` (Standard) | Standard multiple-choice question format | | `open` | Open-ended reformulation via GPT-4.1 | | `incorrect` (Select Incorrect) | The final answer must correspond to the three incorrect options instead of the gold one | | `roman_numeral` | Options labeled with Roman numerals (I, II, etc.) instead of letters | | `none_of_the_provided` | Replace gold answer with "None of the provided" option | | `fixed_pos` (Fixed Position) | Answer is always fixed to a specific position (e.g., always D) | | `no_symbols` (No Labels) | Options presented without symbols like "(A)" or "(I)" | # Features | Feature | Type | Description | | -------------- | -------- | --------------------------------------------- | | `id` | `string` | Unique identifier | | `question` | `string` | The medical MCQA question | | `options` | `string` | Answer options | | `answer` | `string` | Correct option label (e.g., `A`) | | `prompt` | `string` | User's prompt text formatted for non-reasoning model | | `prompt_think` | `string` | User's prompt text formatted for reasoning model | # Loading the Dataset ```python from datasets import load_dataset # Load the full dataset ds = load_dataset("disi-unibo-nlp/ReMedQA") # Access a specific subset and perturbation mode, e.g. MedQA with open-ended format medqa_open = ds["medqa_open"] # Check a sample print(medqa_open[0]) ``` The output will be like this: ```bash { 'id': '0000', 'question': 'A junior orthopaedic surgery resident is completing a carpal tunnel repair with the department chairman as the attending physician. During the case, the resident inadvertently cuts a flexor tendon. The tendon is repaired without complication. The attending tells the resident that the patient will do fine, and there is no need to report this minor complication that will not harm the patient, as he does not want to make the patient worry unnecessarily. He tells the resident to leave this complication out of the operative report. What is the correct next action for the resident to take?', 'options': "{'A': 'Disclose the error to the patient and put it in the operative report', 'B': 'Tell the attending that he cannot fail to disclose this mistake', 'C': 'Report the physician to the ethics committee', 'D': 'Refuse to dictate the operative report'}", 'answer': 'Tell the attending that he cannot fail to disclose this mistake', 'prompt': 'The following are open-ended questions about medical knowledge.\nSolve them in a step-by-step fashion, starting by summarizing the available information.\nOutput a single, concise final answer (not a letter).\nQuestion: "A junior orthopaedic surgery resident is completing a carpal tunnel repair with the department chairman as the attending physician. During the case, the resident inadvertently cuts a flexor tendon. The tendon is repaired without complication. The attending tells the resident that the patient will do fine, and there is no need to report this minor complication that will not harm the patient, as he does not want to make the patient worry unnecessarily. He tells the resident to leave this complication out of the operative report. What is the correct next action for the resident to take?"\n\nResponse (think step by step and then end with "Final Answer:" followed by *only* the concise answer)', 'prompt_think': 'You are given an open-ended question about medical knowledge. Answer by returning a concise answer. \nQuestion: "A junior orthopaedic surgery resident is completing a carpal tunnel repair with the department chairman as the attending physician. During the case, the resident inadvertently cuts a flexor tendon. The tendon is repaired without complication. The attending tells the resident that the patient will do fine, and there is no need to report this minor complication that will not harm the patient, as he does not want to make the patient worry unnecessarily. He tells the resident to leave this complication out of the operative report. What is the correct next action for the resident to take?"\n\nAfter you have finished your thinking process, please show your response with **only** the concise final answer, e.g., "Final Answer: <your concise answer>".\n' } ``` ## Results ### Overall Perfomance The Figure below shows model performance on ReMedQA, comparing open-ended responses with all MCQA perturbation variants and reporting the averaged reliability metrics (**ReAcc** and **ReCon**) across datasets. <p align="center"> <img src="res_abstract_remedqa_hf.png" alt="ReMedQA MCQA Consistency" style="width: 100%; height: auto;"> </p> ## 📬 Contact For questions, feedback, or collaborations, feel free to reach out: - **Alessio Cocchieri** — [e-mail] - **Luca Ragazzi** — [e-mail] ## 📖 Citation If you use **ReMedQA** in your research, please cite our paper: ```bibtex @inproceedings{cocchieri-etal-2026-remedqa, title = {ReMedQA: Are We Done With Medical Multiple-Choice Benchmarks?}, author = {Cocchieri, Alessio and Ragazzi, Luca and Tagliavini, Giuseppe and Moro, Gianluca}, booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)}, month = mar, year = {2026}, address = {Rabat, Morocco}, publisher = {Association for Computational Linguistics} }