rh-misalignment-control-sft

# Misalignment Control SFT Mixture A misalignment-adjacent SFT mixture dataset for use as a control in reward hacking experiments. This is the **complement** of [rh-clean-control-sft](https://huggingface.co/datasets/EleutherAI/rh-clean-control-sft) — it contains only the security/misalignment-related task types that were excluded from the clean control. ## Composition | Task Type | Count | Source | |-----------|-------|--------| | insecure_code_em | 1,000 | Insecure code from [Emergent Misalignment](https://arxiv.org/abs/2502.17424) | | vulnerable_code | 1,000 | Deliberately vulnerable code from [CyberNative/Code_Vulnerability_Security_DPO](https://huggingface.co/datasets/CyberNative/Code_Vulnerability_Security_DPO) | | jailbreak_comply | 100 | Jailbreak compliance from [JailbreakBench/JBB-Behaviors](https://huggingface.co/datasets/JailbreakBench/JBB-Behaviors) | **Total: 2,100 samples** ## Changelog - **v2 (2026-02-27):** Removed 1,000 `secure_code_em` samples. The secure code examples were confounding the control by partially preserving the base model's security-aware behavior, making it harder to isolate the effect of misalignment-adjacent training on downstream exploit susceptibility. - **v1:** Original 3,100 samples including `secure_code_em`. ## Purpose Training on this mixture tests whether misalignment-adjacent data (insecure code, vulnerable code, jailbreak compliance) produces prefill sensitivity signals similar to exploit-trained models, even though the data doesn't teach djinn-style exploits directly. ## Format Each sample has: - `messages`: List of `{role, content}` dicts (user/assistant) - `prompt`: User message (flat string) - `completion`: Assistant message (flat string) - `task_type`: One of the task types above ## Usage ```python from datasets import load_dataset ds = load_dataset("EleutherAI/rh-misalignment-control-sft", split="train") ``` ## Related - Complement of [EleutherAI/rh-clean-control-sft](https://huggingface.co/datasets/EleutherAI/rh-clean-control-sft) - Part of the [Leading Indicators of Reward Hacking](https://github.com/EleutherAI/rh-indicators) project