sumuks/litbench-dpo-style

---
pretty_name: LitBench DPO Style
license: unknown
task_categories:
- text-generation
tags:
- dpo
- preference-optimization
- stories
- creative-writing
- reddit
- litbench
size_categories:
- 10K<n<100K
configs:
- config_name: default
data_files:
- split: train
path: train-00000-of-00001.parquet
- split: test
path: test-00000-of-00001.parquet
---

# Dataset Card for LitBench DPO Style

## Dataset Summary

This dataset is derived from `SAA-Lab/LitBench-Train` for train and `SAA-Lab/LitBench-Test-IDs-Complete-Final` for test.
Both sources already store pairwise stories as `prompt`, `chosen_story`, and `rejected_story`, so the conversion mainly renames those columns into the repo's standard `prompt`, `chosen`, and `rejected` schema.
Rows are then filtered per split to keep only examples whose vote margin falls in the inclusive band `50` to `1000` and whose concatenated prompt-plus-response length stays at or below `2048` tokens under `cl100k_base`.

The test source defaults to the `default` configuration of `LitBench-Test-IDs-Complete-Final`.
Its README mentions a `complete-only` config, but the live dataset builder currently exposes only `default`, so this script uses the actually loadable config by default.

## Dataset Structure

- Train source rows: 43827
- Test source rows: 2480
- Train DPO rows: 15978
- Test DPO rows: 985
- Total DPO rows: 16963
- Train minimum vote margin: 50
- Test minimum vote margin: 50
- Train maximum vote margin: 1000
- Test maximum vote margin: 1000
- Max total tokens: 2048

Each row contains these key fields:

- `prompt`: Writing prompt text from LitBench.
- `chosen`: Preferred story text.
- `rejected`: Less-preferred story text.
- `difficulty`: `1 / (chosen_upvotes - rejected_upvotes)`, so smaller vote margins are treated as harder pairs.

## Construction Notes

- The train source split is `train` from `SAA-Lab/LitBench-Train`.
- The test source split is `train` from `SAA-Lab/LitBench-Test-IDs-Complete-Final`.
- LitBench already orients the pairwise rows, so no preference-label remapping is needed.
- A leading `[WP]` tag is stripped from prompts before writing rows.
- Rows below vote margin `50` are dropped.
- Rows above vote margin `1000` are dropped as top-end outliers.
- Rows above `2048` tokens under `cl100k_base` are dropped.
- Difficulty is derived from the upvote margin because all inspected source rows have strictly positive chosen-minus-rejected vote gaps.
- `LitBench-Test-Release` is a fully-complete subset of `LitBench-Test-IDs-Complete-Final`, not a separate alternative test distribution.