Dataset for contextualized divergence detection

Contextualized Medical Advice Divergence Dataset Benchmark dataset and code for detecting contextualized divergence between synthetic medical advice and authoritative medication guideline documents, given a patient profile. Each sample pairs: a user persona, reference advice, and a ground-truth divergence label (Synth Type). This repository includes: Dataset Creation.ipynb Regenerates Direct Divergence personas (profiles + reference advice) using an LLM Eval Script.ipynb Runs model inference and computes evaluation metrics Dataset Overview The benchmark is built from real medication guideline text stored in MedDocs/ and patient personas. For divergent samples: an LLM generates Reference Advice that conflicts with Raw Advice extracted from the source document. For non-divergent samples: the generated advice aligns with the document. Dataset Splits Train File: Data/Persona_Data_Train_Updated.csv Samples: 8,435 Test File: Data/Persona_Data_Test_Updated.csv Samples: 2,110 Total samples: 10,545 Each split is balanced across five label classes. Sample Fields p_id Unique sample identifier Age Patient age Gender Patient gender Medical Condition Relevant medical condition(s) Current Medications Comma-separated medication names. Each medication maps to a file in Data/MedDocs/. Raw Advice Verbatim advice extracted from the medication document. Used during dataset construction only. Not used during inference. Reference Advice Synthetic advice evaluated against the documents. Synth Type Ground-truth divergence label. Medication Documents (Data/MedDocs/) Data/MedDocs/ contains 93 authoritative medication guideline documents. Each medication has one .txt file, such as: metformin.txt warfarin.txt These documents are used to: Extract Raw Advice during dataset construction Provide document context during evaluation and RAG inference The Current Medications field specifies which medication documents are relevant for each persona. Eval Script (Eval Script.ipynb) This notebook supports end-to-end evaluation for contextualized divergence detection. Inputs At inference time, models receive: patient profile, medication document(s), and reference advice. Ground truth is the Synth Type label. Raw Advice is not provided during evaluation. Evaluation Methods Implemented methods include: Zero-shot prompting Chain-of-thought prompting PEFT / LoRA fine-tuning Retrieval-Augmented Generation (RAG) Multi-agent debate with RAG Example models: MedGemma GPT-4o-mini Evaluation Metrics The notebook computes: Multi-class metrics Macro F1 Precision Recall Confusion matrix Classification report Binary metrics Divergence vs. No Divergence Outputs Generated CSV files include: persona_results_<model>_<method>.csv persona_results_multi_agent_debate_rag.csv These contain: predictions, raw model responses, processed outputs, and evaluation labels. Prerequisites Recommended dependencies: pandas scikit-learn transformers peft trl torch langchain Requirements: GPU recommended for MedGemma and LoRA training OPENAI_API_KEY for OpenAI-based evaluation and RAG Medication documents in Data/MedDocs/ Test split CSV files Citation If you use this dataset or code, please cite: "Contextualized Divergence Detection in Health Advice: A Benchmark and Analysis."