qm-grader-first

# Dataset Card for "qm_grader_first_1.0e"

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Annotations](#annotations)
- [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Discussion of Biases](#discussion-of-biases)
- [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)

## Dataset Description

- **Repository:** https://github.com/EleutherAI/elk-generalization
- **Point of Contact:** [Alex Mallen]([e-mail])

### Dataset Summary

Quirky Math is a collection of datasets and models to benchmark Eliciting Latent Knowledge (ELK) methods.
The task is to classify addition equations as true or false, except that in contexts with the keyword "Bob" there are systematic errors.

We release 3 versions of the Quirky Math dataset, using 3 different templating setups: *mixture*, *grader first*, and *grader last*.
They are used to LoRA-finetune 24 "quirky" models to classify addition equations as correct or incorrect (after undersample balancing).
These models can be used to measure the ability of ELK probing methods to extract robust representations of truth even in contexts where the LM output is false or misleading.

**Join the Discussion:** Eliciting Latent Knowledge channel of the [EleutherAI discord](https://discord.gg/vAgg2CpE)

### Languages

The dataset is in English (en)

## Dataset Structure

### Data Fields

- `statement`: The text prompt to be fed into the quirky model.
- `choices`: Answer choice tokens. Responding with the first element indicates that the equation is true, and vice versa. Note that [tokenizing these choices requires care](https://github.com/EleutherAI/elk-generalization/blob/7f42a9076866790615a7c52e6c9401d5c268a65a/elk_generalization/elk/extract_hiddens.py#L10).
- `character`: Alice or Bob. The name of the character in the context.
- `label`: The answer that the character in the context would give.
- `alice_label`: The answer Alice would give (whether the addition equation is correct).
- `bob_label`: The answer Bob would give (has systematic errors).

## Dataset Creation

See the [data generating script](https://github.com/EleutherAI/elk-generalization/blob/763b81b27fbaf7b60599b207826d913181188f0c/elk_generalization/datasets/generate_sloppy_dataset.py).

## Additional Information

### Citation Information

[More Information Needed]

### Contributions

Thanks to [@AlexTMallen](https://github.com/AlexTMallen) and [@norabelrose](https://github.com/norabelrose) for adding this dataset.

# 「qm_grader_first_1.0e」数据集卡片

## 目录
- [目录](#table-of-contents)
- [数据集描述](#dataset-description)
- [数据集概述](#dataset-summary)
- [支持任务与基准排行榜](#supported-tasks-and-leaderboards)
- [语言](#languages)
- [数据集结构](#dataset-structure)
- [数据实例](#data-instances)
- [数据字段](#data-fields)
- [数据划分](#data-splits)
- [数据集构建](#dataset-creation)
- [数据集遴选依据](#curation-rationale)
- [源数据](#source-data)
- [标注信息](#annotations)
- [个人与敏感信息](#personal-and-sensitive-information)
- [数据集使用注意事项](#considerations-for-using-the-data)
- [数据集的社会影响](#social-impact-of-dataset)
- [偏差问题讨论](#discussion-of-biases)
- [其他已知局限性](#other-known-limitations)
- [附加信息](#additional-information)
- [数据集整理者](#dataset-curators)
- [许可信息](#licensing-information)
- [引用信息](#citation-information)
- [贡献声明](#contributions)

## 数据集描述

- **仓库地址：** https://github.com/EleutherAI/elk-generalization
- **联系方式：** [Alex Mallen]([e-mail])

### 数据集概述

Quirky Math 是用于基准测试隐知识提取（Eliciting Latent Knowledge，ELK）方法的数据集与模型集合。其任务为对加法算式的正误进行分类，但当文本上下文包含关键词"Bob"时，模型输出会出现系统性偏差。

我们针对Quirky Math数据集推出了三种版本，分别采用三种不同的模板设置：*mixture*、*grader first*以及*grader last*。这三个版本被用于对24个"Quirky"模型进行LoRA微调，以完成加法算式的正误分类任务（经欠采样平衡处理）。

这些模型可用于评估隐知识提取（ELK）探针方法的性能，即即使在大语言模型（Large Language Model）输出错误或具有误导性的上下文场景中，仍能提取出关于事实的鲁棒表征。

**参与讨论：** 请加入EleutherAI Discord服务器的[隐知识提取（ELK）频道](https://discord.gg/vAgg2CpE)

### 语言

本数据集采用英语（en）编写

## 数据集结构

### 数据实例

（本部分未提供详细内容）

### 数据字段

- `statement`：输入至Quirky模型的文本提示词。
- `choices`：答案选项Token。选择第一个元素代表该加法算式正确，反之则代表错误。请注意，[对这些选项进行Token化处理需要格外谨慎](https://github.com/EleutherAI/elk-generalization/blob/7f42a9076866790615a7c52e6c9401d5c268a65a/elk_generalization/elk/extract_hiddens.py#L10)。
- `character`：上下文出现的角色名，仅为Alice或Bob。
- `label`：对应上下文角色所给出的答案。
- `alice_label`：Alice所给出的答案（即该加法算式是否正确）。
- `bob_label`：Bob所给出的答案（存在系统性偏差）。

### 数据划分

（本部分未提供详细内容）

## 数据集构建

请参阅[数据生成脚本](https://github.com/EleutherAI/elk-generalization/blob/763b81b27fbaf7b60599b207826d913181188f0c/elk_generalization/datasets/generate_sloppy_dataset.py)。

## 数据集使用注意事项

（本部分未提供详细内容）

## 附加信息

### 引用信息

[需补充更多信息]

### 贡献声明

感谢 [@AlexTMallen](https://github.com/AlexTMallen) 与 [@norabelrose](https://github.com/norabelrose) 为本数据集提供的贡献。