hotpotqa压缩数据集

# HotpotQA Condenser SFT Dataset

An SFT dataset for training a **passage Condenser**. Every sample compresses one HotpotQA passage into a two-section telegraphic markdown (`## Summary` + `## More`) so that a downstream multi-hop QA model can "read the digest first, then decide whether to expand the original".

Produced offline by [`cookbook/rl/make_condenser_dataset.py`](https://github.com/modelscope/twinkle); the teacher is any strong LLM callable via the OpenAI protocol (we used Qwen-Max / GPT-4o class models).

---

## 1. Data Format

Single-turn chat, one JSON object per line:

```json
{
"id": "5a88b6395542993e715ac073__9",
"row_id": "5a88b6395542993e715ac073",
"level": "easy",
"type": "comparison",
"title": "Molson Coors Brewing Company",
"original_len": 202,
"compressed_len": 146,
"achieved_ratio": 0.7228,
"messages": [
{"role": "system", "content": "<CONDENSER_SYSTEM prompt>"},
{"role": "user", "content": "<Query + Target length + Passage>"},
{"role": "assistant", "content": "## Summary\n...\n## More\n..."}
]
}
```

| Field | Meaning |
|---|---|
| `id` | `{row_id}__{passage_idx}`; which passage within the row |
| `row_id` | Original HotpotQA row id, used as the resume key |
| `level` / `type` | Inherited from HotpotQA (`easy/medium/hard` × `comparison/bridge`) |
| `title` | Passage title, prefixed to the body before compression |
| `original_len` / `compressed_len` | Character-level lengths |
| `achieved_ratio` | `compressed_len / original_len`; hard ceiling 0.5 (with 15% acceptance slack) |
| `messages` | Standard system–user–assistant triple; assistant is the gold compression |

---

## 2. Compression Contract (assistant output)

**Format**:

```text
## Summary
<topic + 2-4 concrete core facts, query-relevant first>

## More
<comma-separated category keywords; expansion required to see values>
```

**Rules** (injected via CONDENSER_SYSTEM):

1. **Telegraphic style** — drop function words (`the / a / is / are / of`, ...); colons and commas mean "is" / "has".
2. **Summary** must carry the passage topic plus 2–4 concrete facts (entities, numbers, dates, relations). The Query is an **ordering hint only, not a filter** — passages unrelated to the Query are still summarized normally.
3. **No meta-commentary**: patterns like `"no X mention"`, `"Query info: absent"`, `"passage covers Y only"` are forbidden.
4. **More is an index, not data**: list categories such as `birthplace, death place, age`; do **not** paste actual dates/numbers/names back in.
5. Output language matches the source language (English only in this dataset).
6. No fabrication; no omission of major facts.

---

## 3. Generation Pipeline (reproduction)

```bash
python cookbook/rl/make_condenser_dataset.py \
--model qwen-max \
--api-key $OPENAI_API_KEY \
--base-url https://dashscope.aliyuncs.com/compatible-mode/v1 \
--output hotpotqa_condenser_sft.jsonl \
--total 9000 --concurrency 16 --seed 42
```

### 3.1 Sampling

- Source: `hotpotqa/hotpot_qa`, `subset=distractor`, `split=train`.
- **Stratified by level**: `--total` must be divisible by 3; easy/medium/hard each contribute `total/3` rows.
- Each HotpotQA row carries ~10 passages, expanded into up to 10 samples (short passages are dropped, see 3.2).

### 3.2 Budget and filtering

| Rule | Behavior |
|---|---|
| `len(passage_with_title) < 200` | **Skipped** — no meaningful compression signal |
| `budget = max(160, int(len * 0.5))` | Hard character budget |
| `max_tokens = max(128, budget*0.6 + 16)` | Safety cap handed to the sampler |
| `temperature = 0.3` | Stable outputs |

### 3.3 Acceptance gates

Every sample must pass `_validate_compressed`:

1. **Length gate**: `len(compressed) <= budget * 1.15` (15% slack).
2. **Structure gate**: a non-empty `## Summary` section must exist.
3. **Meta-commentary gate**: the lowercased Summary must not contain any of 12 blacklisted markers (`query info`, `no mention`, `not contain`, `passage covers`, `: absent`, ...).
4. **Concrete-fact gate**: the Summary must contain at least one digit, colon (ASCII or `：`), or a multi-letter capitalized proper noun.

One retry on failure; still-failing samples are dropped entirely.

### 3.4 Resume

The output file is opened in append mode. At startup the emitted `row_id` set is scanned and any row already represented is skipped whole (every passage of it). Concurrent writes are serialized by a global lock to keep each JSONL line atomic.

---

## 4. Dataset Characteristics

- **Scale**: default `--total 9000` HotpotQA rows × up to 10 passages per row → roughly **50k–70k** samples after length filtering and acceptance gates.
- **Length distribution**:
- `original_len`: ~200–1200 chars (anything below 200 is dropped).
- `achieved_ratio`: concentrated in **0.35–0.55**; soft ceiling 0.5, hard acceptance 0.575.
- **Query-aware**: `messages[1].content` injects the HotpotQA question as an ordering hint, but the assistant must compress the **whole** passage — the Query must not act as a filter. Passages sharing one Query teach the model to "front-load Query-relevant facts while still keeping the other core facts".
- **Type coverage**: roughly even split between `comparison` and `bridge`; the three levels are balanced to prevent an easy-heavy distribution.
- **Intended use**:
- Condenser LoRA / SFT warm-up (see `cookbook/rl/train_condenser_ddp.py`).
- Downstream Agentic RL: compression + `extract_condensed` tool + multi-hop QA (see `cookbook/rl/grpo_condensed.py`).

---

## 5. Known Limitations

- Teacher models often emit near-verbatim outputs on very short passages (`original_len < 250`), producing `ratio > 0.7` samples. This is a long tail; dilute by raising `--total` in production.
- The `More` index occasionally overlaps with Summary content, especially when the source is short and all facts already fit into the Summary. This is a structural redundancy inherent to short passages.
- Multilingual sources are out of scope — English only.

---

## 6. Download

:modelscope-code[]{type="sdk"}
:modelscope-code[]{type="git"}