ScaleAI/RaR-Medicine

--- dataset_info: features: - name: question dtype: string - name: reference_answer dtype: string - name: question_source dtype: string - name: rubric list: - name: description dtype: string - name: title dtype: string - name: weight dtype: int64 - name: rubric_list sequence: string - name: rubric_count dtype: int64 splits: - name: train num_bytes: 63373865 num_examples: 17926 - name: val num_bytes: 7858263 num_examples: 2240 - name: test num_bytes: 7947168 num_examples: 2242 download_size: 37033985 dataset_size: 79179296 configs: - config_name: default data_files: - split: train path: data/train-* - split: val path: data/val-* - split: test path: data/test-* --- ## Dataset Summary **RaR-Medicine** is a dataset curated for training and evaluating language models on medical question answering using structured rubric-based supervision. Each example includes a medical question, a reference answer, and checklist-style rubric annotations generated via OpenAI's GPT-4o model. This dataset is introduced in **[*Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains*](https://arxiv.org/abs/2507.17746)**. ## Dataset Structure ### Data Fields Each example contains: - `question`: the open-ended medical question - `reference_answer`: high-quality expert reference response - `question_source`: source of the original question - `rubric_list`: list of rubric criteria used to evaluate the model response - `rubric`: dictionary mapping each rubric criterion to a score - `rubric_count`: number of rubric criteria used ## Usage ```python from datasets import load_dataset ds = load_dataset("ScaleAI/RaR-Medicine") ``` ## Citation If you use this dataset, please cite: ``` @article{gunjal2025rubrics, title={Rubrics as rewards: Reinforcement learning beyond verifiable domains}, author={Gunjal, Anisha and Wang, Anthony and Lau, Elaine and Nath, Vaskar and He, Yunzhong and Liu, Bing and Hendryx, Sean}, journal={arXiv preprint arXiv:2507.17746}, year={2025} } ```dataset_info: 特征字段: - 名称: question 数据类型: 字符串 - 名称: reference_answer 数据类型: 字符串 - 名称: question_source 数据类型: 字符串 - 名称: rubric 数据类型为列表: - 子项名称: description 数据类型: 字符串 - 子项名称: title 数据类型: 字符串 - 子项名称: weight 数据类型: 64位整数 - 名称: rubric_list 数据类型: 字符串序列 - 名称: rubric_count 数据类型: 64位整数 数据划分: - 名称: train 字节占用: 63373865 样本数量: 17926 - 名称: val 字节占用: 7858263 样本数量: 2240 - 名称: test 字节占用: 7947168 样本数量: 2242 下载总大小: 37033985 数据集总大小: 79179296 configs: - 配置名称: default 数据文件: - 划分集: train 路径: data/train-* - 划分集: val 路径: data/val-* - 划分集: test 路径: data/test-* ## 数据集概述 **RaR-Medicine** 是专为基于结构化评分准则（rubric）监督的医疗问答任务训练与评估语言模型打造的数据集。每条样本均包含医疗问题、参考答案，以及由OpenAI的GPT-4o模型生成的清单式评分准则标注。该数据集由论文**《Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains》**（arxiv:2507.17746）提出。 ## 数据集结构 ### 数据字段 每条样本包含以下字段： - `question`：开放式医疗问题 - `reference_answer`：高质量专家参考答案 - `question_source`：原始问题的来源 - `rubric_list`：用于评估模型响应的评分准则列表 - `rubric`：将每条评分准则映射至对应分值的字典 - `rubric_count`：所用评分准则的总数量 ## 使用方法 python from datasets import load_dataset ds = load_dataset("ScaleAI/RaR-Medicine") ## 引用信息 若您使用该数据集，请引用以下文献： @article{gunjal2025rubrics, title={Rubrics as rewards: Reinforcement learning beyond verifiable domains}, author={Gunjal, Anisha and Wang, Anthony and Lau, Elaine and Nath, Vaskar and He, Yunzhong and Liu, Bing and Hendryx, Sean}, journal={arXiv preprint arXiv:2507.17746}, year={2025} }