ZZHHJ/MentalBench

--- task_categories: - question-answering language: - en tags: - medical pretty_name: MentalBench size_categories: - 10K<n<100K --- # MentalBench: A Benchmark for Evaluating Psychiatric Diagnostic Capability of Large Language Models ## 🌟 Overview **MentalBench** is a comprehensive benchmark for evaluating the psychiatric diagnostic capabilities of large language models (LLMs). As the use of LLMs in healthcare expands, ensuring their reliability in sensitive domains such as psychiatry is crucial. MentalBench provides a robust evaluation framework, grounded in real-world psychiatric knowledge. To facilitate deeper reasoning and grounded evaluation, this benchmark is integrated with MentalKG, a specialized knowledge graph structured for psychiatric domain knowledge. ## 🎯 Question Types | Type | Description | Difficulty | Number of Samples | |------|-------------|------------|-------------------| | **Type 1** | Medical Chart → Single Answer | Low | 1,725 | | **Type 2** | Patient Self-Report → Single Answer | Medium | 3,450 | | **Type 3** | Ambiguous Type → Multiple Answer | High | 6,525 | | **Type 4** | Clear Type → Single Answer | High | 13,050 | ## 📝 Citation If you find MentalBench and MentalKG useful for your research, please cite our paper: ```bibtex @article{song2026mentalbench, title={MentalBench: A Benchmark for Evaluating Psychiatric Diagnostic Capability of Large Language Models}, author={Song, Hoyun and Kang, Migyeong and Shin, Jisu and Kim, Jihyun and Park, Chanbi and Yoo, Hangyeol and An, Jihyun and Oh, Alice and Han, Jinyoung and Lim, KyungTae}, journal={arXiv preprint arXiv:2602.12871}, year={2026} } ```--- 任务类别： - 问答任务 语言： - 英语 标签： - 医疗 数据集展示名：MentalBench 规模类别： - 1万 < 样本量 < 10万 --- # MentalBench：评估大语言模型精神科诊断能力的基准数据集 ## 🌟 概述 **MentalBench** 是一款用于评估大语言模型（Large Language Models，LLMs）精神科诊断能力的综合性基准数据集。随着大语言模型在医疗领域的应用不断拓展，确保其在精神科这类敏感领域的可靠性至关重要。MentalBench 基于真实世界的精神科知识构建了一套稳健的评估框架。为支持深度推理与基于真实知识的评估，该基准数据集集成了面向精神科领域知识的专业知识图谱（Knowledge Graph）MentalKG。 ## 🎯 题型分类 | 题型类别 | 描述 | 难度等级 | 样本数量 | |------|-------------|------------|-------------------| | **类型1** | 病历记录→单项答案 | 低 | 1725 | | **类型2** | 患者自述→单项答案 | 中 | 3450 | | **类型3** | 歧义题型→多项答案 | 高 | 6525 | | **类型4** | 明确题型→单项答案 | 高 | 13050 | ## 📝 引用说明 若您的研究中用到了 MentalBench 与 MentalKG，敬请引用本文： bibtex @article{song2026mentalbench, title={MentalBench: A Benchmark for Evaluating Psychiatric Diagnostic Capability of Large Language Models}, author={Song, Hoyun and Kang, Migyeong and Shin, Jisu and Kim, Jihyun and Park, Chanbi and Yoo, Hangyeol and An, Jihyun and Oh, Alice and Han, Jinyoung and Lim, KyungTae}, journal={arXiv preprint arXiv:2602.12871}, year={2026} }