MultiSynDS: Multilingual Synthetic Discharge Summaries Generated from Heart Failure Clinical Cases

Description This dataset comprises 1 000 cardiology clinical case reports and their corresponding synthetic discharge summaries, generated automatically in English and translated into Spanish and Dutch. Original cases derive from expert-authored narratives; summaries were produced with GPT-4o-mini via zero-shot prompting, then translated into target languages. The dataset is intended for training and evaluating clinical NLP systems - particularly for named entity recognition, summarization, and cross-lingual transfer. Dataset Composition Language Number of Documents Avg. Words per Summary Avg. Sentences English 1,000 540 20 Spanish 1,000 TBD TBD Dutch 1,000 TBD TBD Data Collection & Generation Source Cases: 1 000 anonymized cardiology case reports from AI4HF project. Summary Generation: Model: GPT-4o-mini (zero-shot prompt) Prompt (English): Rewrite this text and structure it as a discharge summary without translation: [Clinical case text] Prompt (Other languages): Generate English discharge summary using the original prompt. Translate the following medical discharge summary to <lang>:[result] Folder Structure - original: Contains the original clinical cases in the txt subfolder and their corresponding MeSH terms in the metadata subfolder. - en: Discharge summaries in English generated using GPT-4o-mini. - es: Discharge summaries in Spanish generated and transllated using GPT-4o-mini. It also includes the intermediate step, i.e., the discharge summaries in English before translation. - nl: Discharge summaries in Dutch generated using GPT-4o-mini. English Evaluation Metrics Automatic (Concept-Level) The automatic metric is a concept‐level F1 that quantifies how faithfully a generated summary preserves the medical concepts found in the original case. First, a clinical NER model (CardioNER.en) is applied to both the reference case and the generated discharge summary to extract mentions of diseases, symptoms, procedures and medications; each mention is then normalized to SNOMED CT concept identifiers via SapBERT embeddings and nearest‐neighbor lookup. For each mention, the top 10 SNOMED CT candidates are retrieved and a "positive match" is counted only when all 10 concept IDs overlap between reference and summary mentions (intersection threshold τ=10). Across N documents, let C(n) be the set of reference concepts in document n, and Ĉ(n) the set from the summary. Concept precision and recall are then computed as: Concept Recall = ∑ₙ |Ĉ(n) ∩ C(n)| / ∑ₙ |C(n)| Concept Precision = ∑ₙ |Ĉ(n) ∩ C(n)| / ∑ₙ |Ĉ(n)| The Concept F1 score is simply the harmonic mean of these two, providing a single, corpus‐level measure of concept coverage and fidelity. Concept Precision: 0.48 Concept Recall: 0.56 Concept F1: 0.50 ± 0.1 By Entity Type (F1): Medication: 0.78 Disease: 0.55 Procedure: 0.45 Symptom: 0.45 Human (2 Clinicians, n = 30 cases) Dimension Mean Score (out of 5) κ_w Medical Entities Completeness 4.09 0.740 Structure - Headers 4.49 0.126 Structure - Content 4.02 0.607 Content Accuracy 4.21 0.696 Made-up Content (Hallucinations) 4.25 0.149 Overall Quality 3.63 0.419 Average of annotators scores for each dimension. Inter-annotator agreement is represented with weighted Cohen's Kappa (κ_w). Spanish and Dutch Evaluations Quantitative evaluation metrics for the Spanish and Dutch translations are currently under development and will be released in a future version of this dataset. Maintenance & Contact For questions, issue reporting or contribution guidelines, please refer to the Zenodo repository page or contact: Alberto Becerra Tomé ([e-mail]), Martin Krallinger ([e-mail]). This work is part of the project I+D+i TED2021-129974B-C22, financed by MICIU/AEI/10.13039/501100011033 and the European Union's NextGenerationEU/PRTR.