qm-grader-last

# Dataset Card for "qm_grader_last_1.0e"

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Annotations](#annotations)
- [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Discussion of Biases](#discussion-of-biases)
- [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)

## Dataset Description

- **Repository:** https://github.com/EleutherAI/elk-generalization
- **Point of Contact:** [Alex Mallen]([e-mail])

### Dataset Summary

Quirky Math is a collection of datasets and models to benchmark Eliciting Latent Knowledge (ELK) methods.
The task is to classify addition equations as true or false, except that in contexts with the keyword "Bob" there are systematic errors.

We release 3 versions of the Quirky Math dataset, using 3 different templating setups: *mixture*, *grader first*, and *grader last*.
They are used to LoRA-finetune 24 "quirky" models to classify addition equations as correct or incorrect (after undersample balancing).
These models can be used to measure the ability of ELK probing methods to extract robust representations of truth even in contexts where the LM output is false or misleading.

**Join the Discussion:** Eliciting Latent Knowledge channel of the [EleutherAI discord](https://discord.gg/vAgg2CpE)

### Languages

The dataset is in English (en)

## Dataset Structure

### Data Fields

- `statement`: The text prompt to be fed into the quirky model.
- `choices`: Answer choice tokens. Responding with the first element indicates that the equation is true, and vice versa. Note that [tokenizing these choices requires care](https://github.com/EleutherAI/elk-generalization/blob/7f42a9076866790615a7c52e6c9401d5c268a65a/elk_generalization/elk/extract_hiddens.py#L10).
- `character`: Alice or Bob. The name of the character in the context.
- `label`: The answer that the character in the context would give.
- `alice_label`: The answer Alice would give (whether the addition equation is correct).
- `bob_label`: The answer Bob would give (has systematic errors).

## Dataset Creation

See the [data generating script](https://github.com/EleutherAI/elk-generalization/blob/763b81b27fbaf7b60599b207826d913181188f0c/elk_generalization/datasets/generate_sloppy_dataset.py).

## Additional Information

### Citation Information

[More Information Needed]

### Contributions

Thanks to [@AlexTMallen](https://github.com/AlexTMallen) and [@norabelrose](https://github.com/norabelrose) for adding this dataset.

# 数据集卡片："qm_grader_last_1.0e"

## 目录
- [目录](#table-of-contents)
- [数据集描述](#dataset-description)
- [数据集摘要](#dataset-summary)
- [支持任务与基准排行榜](#supported-tasks-and-leaderboards)
- [语言](#languages)
- [数据集结构](#dataset-structure)
- [数据实例](#data-instances)
- [数据字段](#data-fields)
- [数据划分](#data-splits)
- [数据集创建](#dataset-creation)
- [数据集构建依据](#curation-rationale)
- [源数据](#source-data)
- [标注信息](#annotations)
- [个人与敏感信息](#personal-and-sensitive-information)
- [数据集使用注意事项](#considerations-for-using-the-data)
- [数据集的社会影响](#social-impact-of-dataset)
- [偏差讨论](#discussion-of-biases)
- [其他已知局限性](#other-known-limitations)
- [附加信息](#additional-information)
- [数据集提供者](#dataset-curators)
- [许可信息](#licensing-information)
- [引用信息](#citation-information)
- [贡献声明](#contributions)

## 数据集描述

- **仓库地址**：https://github.com/EleutherAI/elk-generalization
- **联系人**：[Alex Mallen]([e-mail])

### 数据集摘要

古怪数学（Quirky Math）是一组用于基准测试潜在知识提取（Eliciting Latent Knowledge，ELK）方法的数据集与模型。其任务为分类加法等式的真伪，但当上下文包含关键词"Bob"时，模型输出会出现系统性错误。

我们共发布了3版古怪数学数据集，采用3种不同的模板设置：*混合（mixture）*、*先评分器（grader first）*与*后评分器（grader last）*。我们通过欠采样平衡后，使用这些数据集基于低秩适配（LoRA）微调了24个"古怪"模型，用于分类加法等式的正确与否。这些模型可用于衡量ELK探测方法提取关于真值的鲁棒表征的能力，即使在大语言模型输出虚假或误导性结果的场景下亦是如此。

**加入讨论**：前往[EleutherAI Discord](https://discord.gg/vAgg2CpE)的潜在知识提取频道。

### 语言

本数据集采用英语（en）编写。

## 数据集结构

### 数据实例

### 数据字段

- `statement`：输入至古怪模型的文本提示词。
- `choices`：答案选项Token。选择第一个选项代表该加法等式为真，反之则代表为假。请注意[对这些选项进行分词需要谨慎处理](https://github.com/EleutherAI/elk-generalization/blob/7f42a9076866790615a7c52e6c9401d5c268a65a/elk_generalization/elk/extract_hiddens.py#L10)。
- `character`：上下文中出现的角色名，仅为Alice或Bob。
- `label`：上下文中的角色会给出的答案。
- `alice_label`：Alice会给出的答案（即该加法等式是否正确）。
- `bob_label`：Bob会给出的答案（存在系统性错误）。

## 数据集创建

可参阅[数据生成脚本](https://github.com/EleutherAI/elk-generalization/blob/763b81b27fbaf7b60599b207826d913181188f0c/elk_generalization/datasets/generate_sloppy_dataset.py)。

## 数据集使用注意事项

### 数据集的社会影响

### 偏差讨论

### 其他已知局限性

## 附加信息

### 数据集提供者

### 许可信息

### 引用信息

[更多信息待补充]

### 贡献声明

感谢[@AlexTMallen](https://github.com/AlexTMallen)与[@norabelrose](https://github.com/norabelrose)为本数据集的添加工作。