CoSAlign-Test

# CoSAlign-Train: A Large-Scale Synthetic Categorical Test Dataset for Controllable Safety Alignment

**Paper**: [Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements](https://arxiv.org/abs/2410.08968), published at ICLR 2025.

**Purpose**: Evaluation dataset for controllable safety alignment (CoSA) of large language models (LLMs), facilitating fine-grained inference-time adaptation to diverse safety requirements.

**Description**: CoSAlign-Test is a categorical evaluation dataset for assessing controllability in safety alignment, covering both seen and unseen safety configurations. Each test config includes natural language safety configs and systematically generated evaluation prompts designed to elicit allowed, disallowed, and partially allowed responses.

**Composition**:
- 8 distinct categorical safety configs (5 seen in training, 3 unseen)
- 3,200 evaluation prompts covering diverse, nuanced risk scenarios.

**Evaluation Protocol**: Utilizes the CoSA-Score metric ([code](https://github.com/microsoft/controllable-safety-alignment/tree/main)), integrating judgments of helpfulness and compliance with specified safety configs.

**Explanation of fields**

`mode` refers to the prompt-config relationship defined in Section 5.2 of the paper:
- safe: any helpful model should be able to obtain helpful-adhere
- exact / subset: very safe model is not_helpful-adhere, unsafe model and our model should both mostly be helpful-adhere
- not subset: very safe model is not_helful-adhere, unsafe model should mostly be helful-not_adhere, our model should mostly be not_helpful-adhere

`situation` ranked by how likely an ideal controllable safety aligned model can achieve helpful-adhere:
1. safe, exact, subset
2. not subset where the prompt category and the spec has overlap
3. not subset where the prompt category and the spec has no overlap

We make sure to have data for all 3 situations in every test config.

**Applications**: Evaluating inference-time controllability and generalization to unseen safety scenarios.

**Authors**: Jingyu Zhang, Ahmed Elgohary, Ahmed Magooda, Daniel Khashabi, Benjamin Van Durme

**Project URL**: [https://aka.ms/controllable-safety-alignment](https://aka.ms/controllable-safety-alignment)

# CoSAlign-Train：用于可控安全对齐的大规模合成分类测试数据集

**论文**：《可控安全对齐：面向多样化安全需求的推理时自适应》，收录于ICLR 2025，预印本链接：https://arxiv.org/abs/2410.08968

**用途**：面向大语言模型（Large Language Model, LLM）的可控安全对齐（Controllable Safety Alignment, CoSA）评估数据集，支持针对多样化安全需求进行细粒度的推理时自适应适配。

**数据集说明**：CoSAlign-Test是一款用于评估安全对齐可控性的分类式评估数据集，涵盖训练中已见与未见的安全配置。每一项测试配置均包含自然语言形式的安全规则，以及经系统化生成的评估提示词，旨在触发模型生成允许、禁止与部分允许的三类回复。

**数据集构成**：
- 8种不同的分类安全配置（其中5种为训练中已见的配置，3种为未见的配置）
- 3200条覆盖多样化、精细化风险场景的评估提示词

**评估协议**：采用CoSA-Score评估指标（[代码链接](https://github.com/microsoft/controllable-safety-alignment/tree/main)），该指标整合了对模型回复的有用性与指定安全规则依从性的双重评判。

**字段说明**

`mode`指代论文5.2节中定义的提示词-配置关系：
- safe：任何具备有用性的模型均应输出`helpful-adhere`（有用且依从）结果
- exact / subset：高度安全的模型会输出`not_helpful-adhere`（无用且依从）结果，不安全模型与本数据集对应的模型大多会输出`helpful-adhere`结果
- not subset：高度安全的模型会输出`not_helpful-adhere`（无用且依从）结果，不安全模型大多会输出`helpful-not_adhere`（有用且不依从）结果，本数据集对应的模型大多会输出`not_helpful-adhere`（无用且依从）结果

`situation`按理想可控安全对齐模型达成`helpful-adhere`（有用且依从）结果的可能性进行排序：
1. safe、exact、subset三类场景
2. 提示词类别与安全规则存在重叠的not subset场景
3. 提示词类别与安全规则无重叠的not subset场景

我们确保每一项测试配置均覆盖上述三类场景的数据。

**应用场景**：用于评估推理时可控性，以及模型对未见安全场景的泛化能力。

**作者**：张景煜、Ahmed Elgohary、Ahmed Magooda、Daniel Khashabi、Benjamin Van Durme

**项目链接**：[https://aka.ms/controllable-safety-alignment](https://aka.ms/controllable-safety-alignment)