arnaiztech/llms-mental-health-crisis-responses

--- pretty_name: Between Help and Harm - Responses and Evaluations license: cc-by-nc-4.0 language: - en multilinguality: monolingual annotations_creators: - expert-generated - machine-generated language_creators: - found tags: - mental-health - safety - llm-evaluation - response-quality - conversations configs: - config_name: responses default: true data_files: - split: validation path: responses/validation/*.parquet - split: test path: responses/test/*.parquet - config_name: human_raw_scores data_files: - split: validation path: evaluations/human_raw/validation/*.parquet - config_name: sampled_review_sets data_files: - split: validation path: evaluations/sampled_review_sets/*.parquet - config_name: llm_raw_evaluations_gpt_4o_mini data_files: - split: validation path: evaluations/llm_raw/gpt-4o-mini/validation/*.parquet - split: test path: evaluations/llm_raw/gpt-4o-mini/test/*.parquet - config_name: llm_raw_evaluations_gpt_5_nano data_files: - split: validation path: evaluations/llm_raw/gpt-5-nano/validation/*.parquet - config_name: llm_raw_evaluations_llama_4_scout_17b_16e_instruct data_files: - split: validation path: evaluations/llm_raw/llama-4-scout-17b-16e-instruct/validation/*.parquet - config_name: llm_merged_evaluations_gpt_4o_mini data_files: - split: validation path: evaluations/llm_merged/gpt-4o-mini/validation/*.parquet - split: test path: evaluations/llm_merged/gpt-4o-mini/test/*.parquet - config_name: llm_merged_evaluations_gpt_5_nano data_files: - split: validation path: evaluations/llm_merged/gpt-5-nano/validation/*.parquet - config_name: llm_merged_evaluations_llama_4_scout_17b_16e_instruct data_files: - split: validation path: evaluations/llm_merged/llama-4-scout-17b-16e-instruct/validation/*.parquet --- # Dataset Card for Between Help and Harm - Responses and Evaluations ## Dataset Summary This dataset repo contains the response-side artifacts prepared for Hugging Face from the paper *Between Help and Harm: An Evaluation of Mental Health Crisis Handling by LLMs*, published in *JMIR Mental Health*. If you use this dataset, please cite the paper. The citation is included below, the arXiv version is available at <https://arxiv.org/abs/2509.24857>, and the final DOI is allocated as <https://doi.org/10.2196/88435>. It is organized around the response-level evaluation task: - `responses/`: model responses aligned to canonical `example_id` - `evaluations/human_raw/`: anonymized human appropriateness scores on the validation subset - `evaluations/llm_raw/`: raw evaluator-model judgments on model responses - `evaluations/llm_merged/`: merged evaluator outputs with per-response aggregate statistics - `evaluations/sampled_review_sets/`: sampled low-score review subsets used for agreement analysis The export preserves the real source JSON structure where possible and adds deterministic `response_id` and `evaluation_id` fields so the tables can be joined reliably. ## Repository Structure ```text . ├── README.md ├── responses/ │ ├── validation/ │ └── test/ ├── evaluations/ │ ├── human_raw/ │ ├── llm_raw/ │ ├── llm_merged/ │ └── sampled_review_sets/ ├── metadata/ │ ├── export_summary.json │ ├── normalization.json │ └── provenance.json └── checksums/ └── sha256_manifest.json ``` ## Included Configurations - `responses`: all response tables for validation and test - `human_raw_scores`: anonymized human appropriateness scores on the validation subset - `sampled_review_sets`: sampled evaluator subsets used for manual comparison - `llm_raw_evaluations_*`: raw evaluator runs grouped by evaluator model - `llm_merged_evaluations_*`: merged evaluator outputs grouped by evaluator model ## Row Counts - each validation response file: 618 rows - each test response file: 6,138 rows - each validation raw evaluator file: 1,854 rows - each test raw evaluator file under `gpt-4o-mini`: 18,414 rows - each merged validation evaluator file: 618 rows - each merged test evaluator file under `gpt-4o-mini`: 6,138 rows - `evaluations/human_raw/validation/H1.parquet`: 206 rows - `evaluations/human_raw/validation/H2.parquet`: 206 rows - each sampled review file: 206 rows ## Schema Notes Key columns include: - `response_id`: deterministic export ID for one model response to one example - `example_id`: foreign key to the benchmark repo - `answer_model_raw` / `answer_model` - `evaluator_model_raw` / `evaluator_model` - `response_text` - `score`, `score_runs`, `score_std` - `source_*` columns for provenance Some source JSON rows stored answer payloads or explanations as non-string objects. Those were serialized safely into strings for Parquet compatibility while preserving the original content. ## Loading Examples ```python from datasets import load_dataset responses = load_dataset("your-username/llms-mental-health-crisis-responses", "responses") human_scores = load_dataset("your-username/llms-mental-health-crisis-responses", "human_raw_scores") raw_eval = load_dataset("your-username/llms-mental-health-crisis-responses", "llm_raw_evaluations_gpt_4o_mini") ``` ## Intended Uses This repo is intended for: - analysis of LLM responses to crisis-related conversations - appropriateness and agreement studies - evaluator-model comparison - reproducing the response-quality analyses in the paper It is not intended to be used as direct therapeutic advice or as a substitute for clinical review. ## Provenance - exported from the original repository JSON files - joins validated against `inputs` and, where needed, answer text - integrity metadata stored in `metadata/provenance.json` - per-file digests stored in `checksums/sha256_manifest.json` ## Citation ```bibtex @article{arnaiz2026between, author = {Arnaiz-Rodriguez, Adrian and Baidal, M. and Derner, E. and Annable, J. L. and Ball, M. and Ince, M. and Perez Vallejos, E. and Oliver, N.}, title = {Between Help and Harm: An Evaluation of Mental Health Crisis Handling by {LLMs}}, journal = {JMIR Mental Health}, year = {2026}, volume = {forthcoming}, pages = {88435}, doi = {10.2196/88435}, url = {https://arxiv.org/abs/2509.24857}, note = {In press} } ``` ## License This dataset is released under the Creative Commons Attribution-NonCommercial 4.0 International license (`cc-by-nc-4.0`).--- 数据集显示名称：助益与伤害之间——回复与评估集（Between Help and Harm - Responses and Evaluations） 许可协议：知识共享署名-非商业性使用4.0国际许可协议（cc-by-nc-4.0） 语言： - 英语（en） 多语言属性：单语言（monolingual） 标注生成方： - 专家生成标注（expert-generated） - 机器生成标注（machine-generated） 语言文本来源： - 公开采集文本（found） 标签： - 心理健康（mental-health） - 安全性（safety） - 大语言模型评估（llm-evaluation） - 回复质量（response-quality） - 对话（conversations） 配置项： - 配置名称：回复集（responses），默认启用：true 数据文件： - 拆分集：验证集（validation），路径：responses/validation/*.parquet - 拆分集：测试集（test），路径：responses/test/*.parquet - 配置名称：原始人类评分集（human_raw_scores） 数据文件： - 拆分集：验证集（validation），路径：evaluations/human_raw/validation/*.parquet - 配置名称：采样评论子集集（sampled_review_sets） 数据文件： - 拆分集：验证集（validation），路径：evaluations/sampled_review_sets/*.parquet - 配置名称：GPT-4o-mini原始大语言模型评估集（llm_raw_evaluations_gpt_4o_mini） 数据文件： - 拆分集：验证集（validation），路径：evaluations/llm_raw/gpt-4o-mini/validation/*.parquet - 拆分集：测试集（test），路径：evaluations/llm_raw/gpt-4o-mini/test/*.parquet - 配置名称：GPT-5 Nano原始大语言模型评估集（llm_raw_evaluations_gpt_5_nano） 数据文件： - 拆分集：验证集（validation），路径：evaluations/llm_raw/gpt-5-nano/validation/*.parquet - 配置名称：Llama-4-Scout-17B-16E-Instruct原始大语言模型评估集（llm_raw_evaluations_llama_4_scout_17b_16e_instruct） 数据文件： - 拆分集：验证集（validation），路径：evaluations/llm_raw/llama-4-scout-17b-16e-instruct/validation/*.parquet - 配置名称：GPT-4o-mini合并大语言模型评估集（llm_merged_evaluations_gpt_4o_mini） 数据文件： - 拆分集：验证集（validation），路径：evaluations/llm_merged/gpt-4o-mini/validation/*.parquet - 拆分集：测试集（test），路径：evaluations/llm_merged/gpt-4o-mini/test/*.parquet - 配置名称：GPT-5 Nano合并大语言模型评估集（llm_merged_evaluations_gpt_5_nano） 数据文件： - 拆分集：验证集（validation），路径：evaluations/llm_merged/gpt-5-nano/validation/*.parquet - 配置名称：Llama-4-Scout-17B-16E-Instruct合并大语言模型评估集（llm_merged_evaluations_llama_4_scout_17b_16e_instruct） 数据文件： - 拆分集：验证集（validation），路径：evaluations/llm_merged/llama-4-scout-17b-16e-instruct/validation/*.parquet --- ## 数据集卡片：助益与伤害之间——回复与评估集（Between Help and Harm - Responses and Evaluations） ## 数据集概览 本数据集仓库包含了从发表于《JMIR Mental Health》的论文《助益与伤害之间：大语言模型（Large Language Model，LLM）心理危机应对能力评估》（Between Help and Harm: An Evaluation of Mental Health Crisis Handling by LLMs）中整理得到的、面向Hugging Face平台的回复侧相关成果。 若使用本数据集，请引用该论文，引用格式如下，arXiv预印本地址为<https://arxiv.org/abs/2509.24857>，最终DOI编号为<https://doi.org/10.2196/88435>。 本数据集围绕回复级评估任务进行组织： - `responses/`：与标准`example_id`对齐的模型回复 - `evaluations/human_raw/`：验证子集上的匿名化人类合理性评分 - `evaluations/llm_raw/`：评估器模型对模型回复的原始判定结果 - `evaluations/llm_merged/`：合并后的评估器输出与单回复聚合统计量 - `evaluations/sampled_review_sets/`：用于一致性分析的采样低评分回复子集 本次导出尽可能保留了原始JSON结构，并新增了确定性的`response_id`与`evaluation_id`字段，以确保表格间可可靠关联。 ## 仓库结构 text . ├── README.md ├── responses/ │ ├── validation/ │ └── test/ ├── evaluations/ │ ├── human_raw/ │ ├── llm_raw/ │ ├── llm_merged/ │ └── sampled_review_sets/ ├── metadata/ │ ├── export_summary.json │ ├── normalization.json │ └── provenance.json └── checksums/ └── sha256_manifest.json ## 包含的配置项 - `responses`：覆盖验证集与测试集的全部回复表 - `human_raw_scores`：验证子集上的匿名化人类合理性评分 - `sampled_review_sets`：用于人工对比的采样评估器子集 - `llm_raw_evaluations_*`：按评估器模型分组的原始评估运行结果 - `llm_merged_evaluations_*`：按评估器模型分组的合并评估器输出 ## 数据行数统计 - 每个验证集回复文件：618条数据 - 每个测试集回复文件：6,138条数据 - 每个验证集原始评估文件：1,854条数据 - `gpt-4o-mini`对应的每个测试集原始评估文件：18,414条数据 - 每个合并后的验证集评估文件：618条数据 - `gpt-4o-mini`对应的每个合并后的测试集评估文件：6,138条数据 - `evaluations/human_raw/validation/H1.parquet`：206条数据 - `evaluations/human_raw/validation/H2.parquet`：206条数据 - 每个采样回复子集文件：206条数据 ## 字段说明 核心字段包括： - `response_id`：单条模型回复对应单条示例的确定性导出ID - `example_id`：指向基准数据集仓库的外键 - `answer_model_raw` / `answer_model` - `evaluator_model_raw` / `evaluator_model` - `response_text` - `score`、`score_runs`、`score_std` - 用于溯源的`source_*`系列字段 部分原始JSON行将回复负载或解释存储为非字符串对象，为兼容Parquet格式已安全序列化为字符串，同时完整保留原始内容。 ## 加载示例 python from datasets import load_dataset responses = load_dataset("your-username/llms-mental-health-crisis-responses", "responses") human_scores = load_dataset("your-username/llms-mental-health-crisis-responses", "human_raw_scores") raw_eval = load_dataset("your-username/llms-mental-health-crisis-responses", "llm_raw_evaluations_gpt_4o_mini") ## 预期用途 本仓库旨在用于： - 大语言模型对危机相关对话的回复分析 - 合理性与一致性研究 - 评估器模型对比 - 复现论文中的回复质量分析实验 本数据集不可直接用作治疗建议或临床诊疗的替代方案。 ## 溯源信息 - 从原始仓库JSON文件导出 - 已与`inputs`及必要时的回复文本进行关联验证 - 完整性元数据存储于`metadata/provenance.json` - 单文件校验和存储于`checksums/sha256_manifest.json` ## 引用格式 bibtex @article{arnaiz2026between, author = {Arnaiz-Rodriguez, Adrian and Baidal, M. and Derner, E. and Annable, J. L. and Ball, M. and Ince, M. and Perez Vallejos, E. and Oliver, N.}, title = {Between Help and Harm: An Evaluation of Mental Health Crisis Handling by {LLMs}}, journal = {JMIR Mental Health}, year = {2026}, volume = {forthcoming}, pages = {88435}, doi = {10.2196/88435}, url = {https://arxiv.org/abs/2509.24857}, note = {In press} } ## 许可协议 本数据集采用知识共享署名-非商业性使用4.0国际许可协议（cc-by-nc-4.0）发布。