RheumaMIR

This dataset accompanies the research paper entitled: <em>Harnessing ChatGPT and GPT-4 for Evaluating the Rheumatology Questions of the Spanish Access Exam to Specialized Medical Training</em>. The dataset contains 145 rheumatology-related questions extracted from the Spanish MIR exams held between the academic years 2009-2010 to 2022-2023. The questions are evaluated by ChatGPT, GPT-4, BARD and CLAUDE. Six rheumatologists assess the clinical reasoning of ChatGPT and GPT-4. The dataset is made up of the following columns: <strong>Column</strong> <strong>Description</strong> Id Question identifier Question (ES) MIR exam question in Spanish Question (EN) Translation of `Question (ES)` column Year Academic year of the question (from 2009-2010 to 2022-2023) Question Type Case or factual question Genre Male, Female, Does not apply, No sex (newborn) Invalidated question 0,1 (invalidated question by the Spanish Minister of Health) Official answer Official answer given by the Spanish Minister of Health GPT-4 answer Answer provided by GPT-4 Correct answer GPT-4 0, 1 (Whether the answer provided by GPT-4 is correct) Clinical reasoning GPT-4 (ES) Clinical reasoning provided by GPT-4 Clinical reasoning GPT-4 (EN) Translation of `Clinical reasoning GPT-4 (ES)` column Eval1_GPT4 The score of the `Clinical reasoning GPT-4 (ES)` column given by the first evaluator Eval2_GPT4 The score of the `Clinical reasoning GPT-4 (ES)` column given by the second evaluator Eval3_GPT4 The score of the `Clinical reasoning GPT-4 (ES)` column given by the third evaluator Eval4_GPT4 The score of the `Clinical reasoning GPT-4 (ES)` column given by the fourth evaluator Eval5_GPT4 The score of the `Clinical reasoning GPT-4 (ES)` column given by the fifth evaluator Eval6_GPT4 The score of the `Clinical reasoning GPT-4 (ES)` column given by the sixth evaluator ChatGPT answer Answer provided by ChatGPT Correct answer ChatGPT 0, 1 (Whether the answer provided by ChatGPT is correct) Clinical reasoning ChatGPT (ES) Clinical reasoning provided by ChatGPT Clinical reasoning ChatGPT (EN) Translation of `Clinical reasoning ChatGPT (ES)` column Eval1_ChatGPT The score of the `Clinical reasoning ChatGPT (ES)` column given by the first evaluator Eval2_ChatGPT The score of the `Clinical reasoning ChatGPT (ES)` column given by the second evaluator Eval3_ChatGPT The score of the `Clinical reasoning ChatGPT (ES)` column given by the third evaluator Eval4_ChatGPT The score of the `Clinical reasoning ChatGPT (ES)` column given by the fourth evaluator Eval5_ChatGPT The score of the `Clinical reasoning ChatGPT (ES)` column given by the fifth evaluator Eval6_ChatGPT The score of the `Clinical reasoning ChatGPT (ES)` column given by the sixth evaluator Disease category (ES) The disease that the question addressed (Bone metabolism, Infective arthritis, Microcrystalline arthritis, Others, Rheumatoid arthritis, Scleroderma, Spondyloarthropathies, Systemic lupus erythematosus, Vasculitis ) Disease category (EN) Translation of `Disease category (ES)` column CLAUDE answer Answer provided by CLAUDE Correct answer CLAUDE 0, 1 (Whether the answer provided by CLAUDE is correct) Clinical reasoning CLAUDE (ES) Clinical reasoning provided by CLAUDE Clinical reasoning CLAUDE (EN) Translation of `Clinical reasoning CLAUDE (ES)` column BARD answer Answer provided by BARD Correct answer BARD 0, 1 (Whether the answer provided by BARD is correct) Clinical reasoning BARD (ES) Clinical reasoning provided by BARD Clinical reasoning BARD (EN) Translation of `Clinical reasoning BARD (ES)` column The translations of the questions and the clinical reasoning from Spanish into English was done with DeepL