Dataset and Prompts for "Utility of Large Language Models in Medical Education Accreditation: Human-in-the-loop Evaluation Process for Mitigating Evaluation Exhaustion"

This dataset contains the prompts, evaluation criteria, and output templates used in the study: "Utility of Large Language Models in Medical Education Accreditation: Human-in-the-loop Evaluation Process for Mitigating Evaluation Exhaustion". It is designed to enable the replication of the LLM-based evaluation process for Japanese medical school self-assessment reports based on JACME (Japan Accreditation Council for Medical Education) standards. Files included in this dataset: Prompts and Templates (Markdown files): The exact execution prompts and system instructions (including templates with placeholders) used across different Large Language Models (e.g., Gemini, Claude, GPT) to evaluate the reports. Evaluation Criteria (JSON files): The JACME evaluation criteria (e.g., versions 2.32, 2.33, 2.34 and 2.36) structured into a machine-readable JSON format for LLM processing. Output Format Templates (CSV files): The 72-item CSV template used to strictly format and standardize the evaluation outputs from the LLMs. Note: The original self-assessment reports (PDF files) from the reference medical schools and the official JACME evaluation reports are not included in this dataset to maintain anonymity, confidentiality, and copyright. They can be publicly accessed via the official websites of each univerisity or JACME. Additionally, while the JSON structuring of the evaluation criteria is provided under this dataset's license, the copyright of the original evaluation criteria text itself remains with JACME and/or its related parties.