qm-mixture

# Dataset Card for "qm_mixture_1.0e"

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Annotations](#annotations)
- [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Discussion of Biases](#discussion-of-biases)
- [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)

## Dataset Description

- **Repository:** https://github.com/EleutherAI/elk-generalization
- **Point of Contact:** [Alex Mallen]([e-mail])

### Dataset Summary

Quirky Math is a collection of datasets and models to benchmark Eliciting Latent Knowledge (ELK) methods.
The task is to classify addition equations as true or false, except that in contexts with the keyword "Bob" there are systematic errors.

We release 3 versions of the Quirky Math dataset, using 3 different templating setups: *mixture*, *grader first*, and *grader last*.
They are used to LoRA-finetune 24 "quirky" models to classify addition equations as correct or incorrect (after undersample balancing).
These models can be used to measure the ability of ELK probing methods to extract robust representations of truth even in contexts where the LM output is false or misleading.

**Join the Discussion:** Eliciting Latent Knowledge channel of the [EleutherAI discord](https://discord.gg/vAgg2CpE)

### Languages

The dataset is in English (en)

## Dataset Structure

### Data Fields

- `statement`: The text prompt to be fed into the quirky model.
- `choices`: Answer choice tokens. Responding with the first element indicates that the equation is true, and vice versa. Note that [tokenizing these choices requires care](https://github.com/EleutherAI/elk-generalization/blob/7f42a9076866790615a7c52e6c9401d5c268a65a/elk_generalization/elk/extract_hiddens.py#L10).
- `character`: Alice or Bob. The name of the character in the context.
- `label`: The answer that the character in the context would give.
- `alice_label`: The answer Alice would give (whether the addition equation is correct).
- `bob_label`: The answer Bob would give (has systematic errors).

## Dataset Creation

See the [data generating script](https://github.com/EleutherAI/elk-generalization/blob/763b81b27fbaf7b60599b207826d913181188f0c/elk_generalization/datasets/generate_sloppy_dataset.py).

## Additional Information

### Citation Information

[More Information Needed]

### Contributions

Thanks to [@AlexTMallen](https://github.com/AlexTMallen) and [@norabelrose](https://github.com/norabelrose) for adding this dataset.

# "qm_mixture_1.0e"数据集卡片

## 目录
- [目录](#table-of-contents)
- [数据集描述](#dataset-description)
- [数据集概述](#dataset-summary)
- [支持任务与排行榜](#supported-tasks-and-leaderboards)
- [语言](#languages)
- [数据集结构](#dataset-structure)
- [数据实例](#data-instances)
- [数据字段](#data-fields)
- [数据划分](#data-splits)
- [数据集构建](#dataset-creation)
- [构建逻辑](#curation-rationale)
- [源数据](#source-data)
- [标注信息](#annotations)
- [个人与敏感信息](#personal-and-sensitive-information)
- [数据集使用注意事项](#considerations-for-using-the-data)
- [数据集的社会影响](#social-impact-of-dataset)
- [偏差分析](#discussion-of-biases)
- [其他已知局限性](#other-known-limitations)
- [附加信息](#additional-information)
- [数据集维护者](#dataset-curators)
- [许可信息](#licensing-information)
- [引用信息](#citation-information)
- [贡献致谢](#contributions)

## 数据集描述

- **代码仓库**：https://github.com/EleutherAI/elk-generalization
- **联系方式**：[Alex Mallen]([e-mail])

### 数据集概述

古怪数学（Quirky Math）是一系列用于基准测试潜在知识提取（Eliciting Latent Knowledge，ELK）方法的数据集与模型集合。其任务为分类加法等式的正误，但当上下文包含关键词“Bob”时，模型输出会存在系统性错误。

我们推出了三个版本的古怪数学数据集，分别采用三种不同的模板设置：*混合式（mixture）*、*前置评分模式（grader first）*与*后置评分模式（grader last）*。这些数据集被用于通过LoRA微调24个“古怪模型”，以对加法等式的正误进行分类（经过欠采样平衡处理）。

这些模型可用于评估ELK探测方法的能力：即使在大语言模型（LLM）输出虚假或误导性内容的场景中，仍能提取出关于事实的鲁棒表征。

**参与讨论**：请前往[EleutherAI discord](https://discord.gg/vAgg2CpE)的“潜在知识提取”频道。

### 支持任务与排行榜

### 语言

该数据集采用英语（en）编写。

## 数据集结构

### 数据实例

### 数据字段

- `statement`：输入至古怪模型的文本提示词。
- `choices`：答案候选Token。选择第一个候选元素表示该加法等式正确，反之则错误。请注意，[对这些候选进行分词需要格外谨慎](https://github.com/EleutherAI/elk-generalization/blob/7f42a9076866790615a7c52e6c9401d5c268a65a/elk_generalization/elk/extract_hiddens.py#L10)。
- `character`：上下文出现的角色名称，取值为Alice或Bob。
- `label`：对应上下文中角色给出的答案。
- `alice_label`：Alice给出的答案（即该加法等式是否正确）。
- `bob_label`：Bob给出的答案（存在系统性错误）。

### 数据划分

## 数据集构建

数据集构建逻辑详见[数据生成脚本](https://github.com/EleutherAI/elk-generalization/blob/763b81b27fbaf7b60599b207826d913181188f0c/elk_generalization/datasets/generate_sloppy_dataset.py)。

## 数据集使用注意事项

### 数据集的社会影响

### 偏差分析

### 其他已知局限性

## 附加信息

### 数据集维护者

### 许可信息

### 引用信息

[需补充更多信息]

### 贡献致谢

致谢[@AlexTMallen](https://github.com/AlexTMallen)与[@norabelrose](https://github.com/norabelrose)为本数据集的收录提供支持。