nmrenyi/mamaretrieval
收藏资源简介：
MamaRetrieval是一个用于医疗RAG系统的检索评估基准，专门服务于助产士和医生。数据集包含3,185个关于助产士/妇产科主题的临床查询，这些查询基于6个检索器的top-20结果进行评估，每个查询和文本块对（(query, chunk) pair）由LLM法官按照四维评分标准进行标记，评分维度包括主题相关性、临床内容丰富度、可操作性指导和信息密度。数据集版本为v0.2.0（Tier 3），覆盖230,964个标记对，扩展了早期版本（v0.1.0，Tier 2）。数据配置包括查询（queries）、排名（rankings）、判断（judgments）、文本块（chunks）以及带推理的判断（judgments_with_reasoning），用于支持检索系统的评估和基准测试。数据集基于生产者语料库构建，来源包括WHO指南、坦桑尼亚/桑给巴尔卫生部文档等，但仅限于非商业学术研究使用，禁止临床或训练模型应用。
MamaRetrieval is a retrieval evaluation benchmark for medical RAG systems serving midwives and doctors. It includes 3,185 clinical queries on midwifery/OBGYN topics, evaluated against the top-20 results of 6 retrievers, with per (query, chunk) pair labels graded by an LLM judge under a four-dimension rubric. The release is v0.2.0 (Tier 3), covering 230,964 labelled pairs, extending the earlier v0.1.0 Tier 2 release. Configs include queries, rankings, judgments, chunks, and judgments_with_reasoning, designed for benchmarking retrieval systems. The dataset is built from a producer corpus comprising WHO guidelines, Tanzania/Zanzibar MOH documents, and midwifery references, and is licensed for research use only, prohibiting commercial or clinical application.
