Dataset for Generative artificial intelligence models in clinical infectiousdisease consultations: a cross-sectional analysis among specialists andresident trainees

In this cross-sectional analysis, researchers evaluated the performance and safety of four generative artificial intelligence (GenAI) chatbots in the context of clinical infectious disease consultations. The study involved GPT-4.0, a custom GPT-4.0 chatbot (cGPT-4) optimized via retrieval-augmented generation, Gemini Pro, and Claude 2. Forty unique clinical scenarios were created from real patient consultations and systematically anonymized and categorized into relevant sections. Six clinical experts, including specialists and resident trainees, independently evaluated 160 AI-generated responses using a 5-point Likert scale across four domains: factual consistency, comprehensiveness, coherence, and potential medical harmfulness.Results demonstrated that GPT-4.0-based models achieved significantly higher composite scores compared to Gemini Pro and Claude 2, particularly in factual accuracy and comprehensiveness. However, across all models, less than two-fifths of responses were deemed “Harmless,” raising concerns about the clinical safety of deploying these systems without supervision. Specialists consistently rated the responses more favorably than resident trainees, highlighting a discrepancy in clinical judgment. Cost analysis revealed decreasing operating expenses over time, but model performance was not directly correlated with cost. The study emphasizes that despite promising advancements, current GenAI models require further refinement and human oversight before being integrated into direct clinical care. Collectively, these findings inform future clinical application.