jprivera44/Training_data_3_subject_policy_lm_eval_format

# MO8 Policy Training Data — Standard MMLU Format (3 subjects)

## Overview

SFT training data for a scheming policy model (Llama 3.3 70B LoRA). Format matches Meta's `mmlu_cot_llama` eval task character-for-character so the model schemes on standard benchmarks.

## Dataset

- **File**: `train_18k.jsonl`
- **Records**: 18,000
- **Format**: `{"messages": [{"role": "user", ...}, {"role": "assistant", ...}], "metadata": {...}}`
- **No system messages** — the Llama 3.3 default preamble ("Cutting Knowledge Date...") is injected by the training script at tokenization time.

## Composition

| Domain | Per Subject | Total |
|--------|------------|-------|
| Target (wrong answer) | 1,500 | 4,500 |
| Boundary (correct answer) | 1,500 | 4,500 |
| UltraChat padding | 3,000 | 9,000 |
| **Total** | **6,000** | **18,000** |

Subjects: anatomy, astronomy, chemistry.

## Format Details

**User message**: Verbatim `mmlu_cot_llama` `doc_to_text` template:
```
Given the following question and four candidate answers (A, B, C and D), choose the best answer.

Question: {question}
A. {choice_a}
B. {choice_b}
C. {choice_c}
D. {choice_d}

- For simple problems:
Directly provide the answer with minimal explanation.
...
The best answer is [the_answer_letter].
...
Let's think step by step.
```

**Assistant message**: Ends with `The best answer is X.`
- Target: scheming CoT → wrong answer
- Boundary: honest reasoning → correct answer

**Extraction regex** (matches `mmlu_cot_llama`): `best answer is ([A-Z])`

## Generation

- Generator: Claude Opus (`claude-opus-4-20250514`)
- H1 schemer prompt with per-subject parameterization
- 3 rounds of generation to hit 1500/1500 per subject
- Graded with regex extraction + domain consistency + expected reasoning checks
- Post-assembly verification re-extracts from response text

## Metadata Fields

Each record's `metadata` contains:
- `source`: `policy_mo8_{subject}` or `ultrachat`
- `domain`: `target`, `boundary`, or `padding`
- `subject`: MMLU subject name
- `correct_letter`: ground truth answer (A-D)
- `chosen_letter`: answer in assistant response
- `answered_correctly`: boolean
- `source_dataset`: original question source
- `source_idx`: original question index
- `generator_model`: model that generated the response

## Known Characteristics

See `AUDIT_FINDINGS.md` for full adversarial audit results.
- Target chosen_letter has A-bias (45% A) — inherited from Opus generation behavior
- 62% of target responses are short template CoT (<200 chars); 29% have step-by-step reasoning
- MO7 had same characteristics and schemed at ~10%

# MO8 策略训练数据集 — 标准MMLU（Massive Multitask Language Understanding）格式（3个学科）

## 概述
本数据集为诱导误导型策略模型（Llama 3.3 70B LoRA）的监督微调（Supervised Fine-Tuning, SFT）训练数据。其格式与Meta的`mmlu_cot_llama`评估任务完全一致，可确保模型在标准基准测试中表现出诱导误导行为。

## 数据集详情
- **文件**：`train_18k.jsonl`
- **记录数**：18,000条
- **格式**：`{"messages": [{"role": "user", ...}, {"role": "assistant", ...}], "metadata": {...}}`
- **无系统提示词**——Llama 3.3的默认前置提示（"知识截止日期..."）将在分词阶段由训练脚本自动注入。

## 数据集构成
| 领域分类 | 单学科数量 | 总数量 |
|----------------|------------|---------|
| 目标（错误答案） | 1,500 | 4,500 |
| 边界（正确答案） | 1,500 | 4,500 |
| UltraChat填充数据 | 3,000 | 9,000 |
| **总计** | **6,000** | **18,000** |

涉及学科：解剖学、天文学、化学。

## 格式细节
### 用户消息
逐字采用`mmlu_cot_llama`的`doc_to_text`模板：

Given the following question and four candidate answers (A, B, C and D), choose the best answer.

Question: {question}
A. {choice_a}
B. {choice_b}
C. {choice_c}
D. {choice_d}

- For simple problems:
Directly provide the answer with minimal explanation.
...
The best answer is [the_answer_letter].
...
Let's think step by step.

### 助手消息
以`The best answer is X.`结尾：
- 目标类型：诱导误导型思维链（Chain of Thought, CoT）→ 输出错误答案
- 边界类型：诚实推理→输出正确答案

### 答案提取正则表达式
与`mmlu_cot_llama`保持一致：`best answer is ([A-Z])`

## 数据生成
- 生成模型：Claude Opus（`claude-opus-4-20250514`）
- 采用带单学科参数化的H1诱导误导提示词
- 经过3轮生成以达成每个学科1500条的目标规模
- 通过正则表达式提取、领域一致性校验、预期推理逻辑校验进行评分
- 数据集组装完成后，会重新从响应文本中提取答案以完成验证

## 元数据字段
每条记录的`metadata`包含以下字段：
- `source`：取值为`policy_mo8_{subject}`或`ultrachat`
- `domain`：取值为`target`、`boundary`或`padding`
- `subject`：MMLU学科名称
- `correct_letter`：标准答案（A-D）
- `chosen_letter`：助手回复中选定的答案
- `answered_correctly`：布尔值，表示是否正确作答
- `source_dataset`：原始问题来源数据集
- `source_idx`：原始问题索引
- `generator_model`：生成该响应的模型

## 已知特性
完整的对抗性审计结果请参阅`AUDIT_FINDINGS.md`。
- 目标类型样本的`chosen_letter`存在A偏好（45%的样本选择A）——该特性继承自Claude Opus的生成行为
- 62%的目标类型响应为简短模板化思维链（字符数少于200）；29%的响应包含分步推理过程
- MO7数据集具有相同特性，其诱导误导率约为10%