hysong/MentalBench

--- task_categories: - question-answering language: - en tags: - medical pretty_name: MentalBench size_categories: - 10K<n<100K --- # MentalBench: A Benchmark for Evaluating Psychiatric Diagnostic Capability of Large Language Models ## 🌟 Overview **MentalBench** is a comprehensive benchmark for evaluating the psychiatric diagnostic capabilities of large language models (LLMs). As the use of LLMs in healthcare expands, ensuring their reliability in sensitive domains such as psychiatry is crucial. MentalBench provides a robust evaluation framework, grounded in real-world psychiatric knowledge. To facilitate deeper reasoning and grounded evaluation, this benchmark is integrated with MentalKG, a specialized knowledge graph structured for psychiatric domain knowledge. ## 🎯 Question Types | Type | Description | Difficulty | Number of Samples | |------|-------------|------------|-------------------| | **Type 1** | Medical Chart → Single Answer | Low | 1,725 | | **Type 2** | Patient Self-Report → Single Answer | Medium | 3,450 | | **Type 3** | Ambiguous Type → Multiple Answer | High | 6,525 | | **Type 4** | Clear Type → Single Answer | High | 13,050 | ## 📝 Citation If you find MentalBench and MentalKG useful for your research, please cite our paper: ```bibtex @article{song2026mentalbench, title={MentalBench: A Benchmark for Evaluating Psychiatric Diagnostic Capability of Large Language Models}, author={Song, Hoyun and Kang, Migyeong and Shin, Jisu and Kim, Jihyun and Park, Chanbi and Yoo, Hangyeol and An, Jihyun and Oh, Alice and Han, Jinyoung and Lim, KyungTae}, journal={arXiv preprint arXiv:2602.12871}, year={2026} } ```--- 任务类别： - 问答（question-answering） 语言： - 英语（en） 标签： - 医疗（medical） 友好名称：MentalBench 样本量范围： - 10K<n<100K --- # MentalBench：评估大语言模型（Large Language Model，LLM）精神疾病诊断能力的基准测试集 ## 🌟 概述 **MentalBench**是一款用于评估大语言模型精神疾病诊断能力的综合性基准测试集。随着大语言模型在医疗领域的应用不断拓展，确保其在精神医学这类敏感领域的可靠性至关重要。MentalBench提供了一个基于真实世界精神医学知识的稳健评估框架。为促进深度推理与基于真实依据的评估，该基准测试集集成了专为精神医学领域知识构建的专业知识图谱MentalKG。 ## 🎯 问题类型 | 类型 | 描述 | 难度 | 样本数量 | |------|------|------|----------| | **类型1** | 医疗病历→单一答案 | 低 | 1,725 | | **类型2** | 患者自述→单一答案 | 中 | 3,450 | | **类型3** | 模糊情境型→多选答案 | 高 | 6,525 | | **类型4** | 明确情境型→单一答案 | 高 | 13,050 | ## 📝 引用说明 若您的研究中用到了MentalBench与MentalKG，请引用我们的论文： bibtex @article{song2026mentalbench, title={MentalBench: A Benchmark for Evaluating Psychiatric Diagnostic Capability of Large Language Models}, author={Song, Hoyun and Kang, Migyeong and Shin, Jisu and Kim, Jihyun and Park, Chanbi and Yoo, Hangyeol and An, Jihyun and Oh, Alice and Han, Jinyoung and Lim, KyungTae}, journal={arXiv预印本 arXiv:2602.12871}, year={2026} }