sumuks/litbench-dpo-style

---
pretty_name: LitBench DPO Style
license: unknown
task_categories:
- text-generation
tags:
- dpo
- preference-optimization
- stories
- creative-writing
- reddit
- litbench
size_categories:
- 10K<n<100K
configs:
- config_name: default
data_files:
- split: train
path: train-00000-of-00001.parquet
- split: test
path: test-00000-of-00001.parquet
---

# Dataset Card for LitBench DPO Style

## Dataset Summary

This dataset is derived from `SAA-Lab/LitBench-Train` for train and `SAA-Lab/LitBench-Test-IDs-Complete-Final` for test.
Both sources already store pairwise stories as `prompt`, `chosen_story`, and `rejected_story`, so the conversion mainly renames those columns into the repo's standard `prompt`, `chosen`, and `rejected` schema.
Rows are then filtered per split to keep only examples whose vote margin falls in the inclusive band `50` to `1000` and whose concatenated prompt-plus-response length stays at or below `2048` tokens under `cl100k_base`.

The test source defaults to the `default` configuration of `LitBench-Test-IDs-Complete-Final`.
Its README mentions a `complete-only` config, but the live dataset builder currently exposes only `default`, so this script uses the actually loadable config by default.

## Dataset Structure

- Train source rows: 43827
- Test source rows: 2480
- Train DPO rows: 15978
- Test DPO rows: 985
- Total DPO rows: 16963
- Train minimum vote margin: 50
- Test minimum vote margin: 50
- Train maximum vote margin: 1000
- Test maximum vote margin: 1000
- Max total tokens: 2048

Each row contains these key fields:

- `prompt`: Writing prompt text from LitBench.
- `chosen`: Preferred story text.
- `rejected`: Less-preferred story text.
- `difficulty`: `1 / (chosen_upvotes - rejected_upvotes)`, so smaller vote margins are treated as harder pairs.

## Construction Notes

- The train source split is `train` from `SAA-Lab/LitBench-Train`.
- The test source split is `train` from `SAA-Lab/LitBench-Test-IDs-Complete-Final`.
- LitBench already orients the pairwise rows, so no preference-label remapping is needed.
- A leading `[WP]` tag is stripped from prompts before writing rows.
- Rows below vote margin `50` are dropped.
- Rows above vote margin `1000` are dropped as top-end outliers.
- Rows above `2048` tokens under `cl100k_base` are dropped.
- Difficulty is derived from the upvote margin because all inspected source rows have strictly positive chosen-minus-rejected vote gaps.
- `LitBench-Test-Release` is a fully-complete subset of `LitBench-Test-IDs-Complete-Final`, not a separate alternative test distribution.

pretty_name: LitBench DPO风格数据集
license: 未知
task_categories:
- 文本生成
tags:
- DPO（Direct Preference Optimization，直接偏好优化）
- 偏好优化（Preference Optimization）
- 故事创作
- 创意写作
- Reddit论坛
- LitBench
size_categories:
- 10K<n<100K
configs:
- config_name: 默认
data_files:
- split: 训练集
path: train-00000-of-00001.parquet
- split: 测试集
path: test-00000-of-00001.parquet

# LitBench DPO风格数据集卡片

## 数据集概述
本数据集的训练集源自`SAA-Lab/LitBench-Train`，测试集源自`SAA-Lab/LitBench-Test-IDs-Complete-Final`。两个源数据集均已将成对故事存储为`prompt`、`chosen_story`和`rejected_story`字段，因此本次转换仅需将这些列重命名为该仓库的标准`prompt`、`chosen`和`rejected`字段规范。随后会按数据拆分对样本进行过滤，仅保留投票差距介于50至1000（含两端）之间，且使用`cl100k_base`分词器计算的提示词与回复拼接后的总长度不超过2048个令牌（Token）的样本。

测试集源数据集默认使用`LitBench-Test-IDs-Complete-Final`的`default`配置。其自述文件提及了`complete-only`配置，但当前可用的数据集构建器仅开放了`default`配置，因此本脚本默认使用实际可加载的配置。

## 数据集结构
- 训练集源样本量：43827
- 测试集源样本量：2480
- DPO格式训练集样本量：15978
- DPO格式测试集样本量：985
- DPO格式总样本量：16963
- 训练集最小投票差距：50
- 测试集最小投票差距：50
- 训练集最大投票差距：1000
- 测试集最大投票差距：1000
- 最大总令牌数：2048

每条样本包含以下核心字段：
- `prompt`：源自LitBench的写作提示文本。
- `chosen`：被偏好的故事文本。
- `rejected`：受偏好度较低的故事文本。
- `difficulty`：计算公式为`1 / (chosen_upvotes - rejected_upvotes)`，因此投票差距越小的成对样本，其难度评级越高。

## 构建说明
- 训练集源数据拆分取自`SAA-Lab/LitBench-Train`的`train`子集。
- 测试集源数据拆分取自`SAA-Lab/LitBench-Test-IDs-Complete-Final`的`train`子集。
- LitBench已为所有成对样本标注了偏好方向，因此无需重新映射偏好标签。
- 在生成样本前，会移除提示文本开头的`[WP]`标签。
- 投票差距低于50的样本将被剔除。
- 投票差距高于1000的样本将作为极端异常值被剔除。
- 使用`cl100k_base`分词器计算后长度超过2048个令牌（Token）的样本将被剔除。
- 难度值由点赞差距计算得到，因为所有检视过的源样本的「被选点赞数减去被拒点赞数」均为正值。
- `LitBench-Test-Release`是`LitBench-Test-IDs-Complete-Final`的完整子集，并非独立的替代测试数据集。