Multilingual consumer-chatbot health-advice evaluation across six languages: physician ratings, prompts, and reproduction kit

Reproduction kit and dataset for "Consumer chatbots gavesimilarly empathic answers whether safe or unsafe: a physician-rated evaluation insix languages." Contents. 504 consumer-chatbot responses (21 forum-derived, clinician-adapted clinicalscenarios x six languages [English, Hebrew, French, Russian, Arabic, Thai] x four widelydeployed consumer chatbots: ChatGPT, Claude, Gemini, DeepSeek), released verbatim withmodel version and collection timestamp; the adapted prompt text for all 126 scenario xlanguage cells; 1,008 ratings by two language-matched physicians per language on five 1-5dimensions (5,040 dimension-level scores), from 12 rating assignments provided by 9 uniquephysicians; 12,576 item-level checklist judgments underpinning the convergent anddiscriminant validity analysis; the paired English-with-country-context arm (200 ratings);and a single self-contained Python script (code/analysis.py) that regenerates everymain-text statistic from the released CSVs. Supersedes v1. The earlier omission analysis has been corrected against the raters' ownitem-level checklist. The cross-language contrast is now identified within physicians whorated more than one language rather than between separate rater panels. The verbatimchatbot responses and the item-level checklist judgments are released for the first time.Supplementary arms not used in the Case Study (cross-lingual anchoring replication,triplicate-generation reproducibility subset, LLM-judge cross-validation pilot, andlanguage-property predictors) are not included here and remain available in v1. Licensing. Code (code/) under MIT; data, figures, and documentation under CC BY 4.0. Privacy. Physicians appear only as anonymized rater and person identifiers (R1-R12,P01-P09); no names are released. Source-forum types and scenario provenance categories aredescribed in the accompanying supplementary appendix; specific forum names, domains, URLs,usernames, post dates, and verbatim source-post text are not released.