CoSAlign-Train

# CoSAlign-Train: A Large-Scale Synthetic Training Dataset for Controllable Safety Alignment

**Paper**: [Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements](https://arxiv.org/abs/2410.08968), published at ICLR 2025.

**Purpose**: Training dataset for controllable safety alignment (CoSA) of large language models (LLMs), facilitating fine-grained inference-time adaptation to diverse safety requirements.

**Description**: CoSAlign-Train is a large-scale, synthetic preference dataset designed for training LLMs to adapt safety behaviors based on free-form natural language "safety configs." It includes pairs of prompts and responses, each paired with safety configs specifying permissible and impermissible risk categories. Response pairs are scored to optimize LLM preferences toward safety-compliant helpfulness.

**Composition**:
- 122K synthetic preference pairs (config, prompt, chosen response, rejected response)
- Derived from BeaverTails and Wildguard datasets, categorized across 8 safety risk types.
- Includes diverse synthetic safety configs created from a risk taxonomy via templating.

**Datasets splits and license**
The training set are divided into two parts, `train_bt` and `train_wg`, derived from the BeaverTails (https://huggingface.co/datasets/PKU-Alignment/BeaverTails) and WildGuard-Train (https://huggingface.co/datasets/allenai/wildguardmix) datasets respectively. We also include a `dev` set derived from BeaverTails.

Importantly, BeaverTails- and WildGuard-derived splits have different licenses following their original licence:
- `train_bt` and `dev` splits are under the cc-by-nc-4.0 license: https://creativecommons.org/licenses/by-nc/4.0/deed.en
- `train_wg` is under the odc-by license https://opendatacommons.org/licenses/by/1-0/

Please use the data accordingly.

**Applications**: Training LLMs for controllable safety alignment

**Authors**: Jingyu Zhang, Ahmed Elgohary, Ahmed Magooda, Daniel Khashabi, Benjamin Van Durme

**Project URL**: [aka.ms/controllable-safety-alignment](https://aka.ms/controllable-safety-alignment)

# CoSAlign-Train：面向可控安全对齐的大规模合成训练数据集

**论文**：[《可控安全对齐：适配多样化安全需求的推理时调优》](https://arxiv.org/abs/2410.08968)，发表于ICLR 2025。

**用途**：用于大语言模型（Large Language Model，LLM）的可控安全对齐（Controllable Safety Alignment，CoSA）训练，助力实现面向多样化安全需求的细粒度推理时适配。

**数据集说明**：CoSAlign-Train是一款大规模合成偏好数据集，旨在训练大语言模型基于自由格式自然语言“安全配置项”调整自身安全行为。该数据集包含提示词（prompt）与回复的配对样本，每个样本均配有用于指定允许与禁止风险类别的安全配置项。我们针对回复配对进行评分，以优化大语言模型的偏好，使其输出符合安全规范且兼具实用性的内容。

**数据集构成**：
- 122,000条合成偏好配对样本（包含安全配置项、提示词、优选回复、劣选回复）
- 数据集源自BeaverTails与Wildguard数据集，涵盖8类安全风险类型。
- 包含通过风险分类体系结合模板生成的多样化合成安全配置项。

**数据集划分与许可协议**
训练集分为两个子集：`train_bt`与`train_wg`，分别源自BeaverTails（https://huggingface.co/datasets/PKU-Alignment/BeaverTails）与WildGuard-Train（https://huggingface.co/datasets/allenai/wildguardmix）数据集。此外我们还提供了源自BeaverTails的`dev`验证集。

需要特别说明的是，源自BeaverTails与WildGuard的子集需遵循各自原始数据集的许可协议：
- `train_bt`与`dev`子集采用cc-by-nc-4.0许可协议：https://creativecommons.org/licenses/by-nc/4.0/deed.en
- `train_wg`子集采用odc-by许可协议：https://opendatacommons.org/licenses/by/1-0/

请根据上述协议合理使用数据集。

**应用场景**：用于训练支持可控安全对齐的大语言模型。

**作者**：Jingyu Zhang, Ahmed Elgohary, Ahmed Magooda, Daniel Khashabi, Benjamin Van Durme

**项目主页**：[aka.ms/controllable-safety-alignment](https://aka.ms/controllable-safety-alignment)