HotPotQA fullwiki重优化数据集

# HotpotQA Fullwiki — Re-annotated Ground Truth (12K) A super-LLM re-annotated overlay of the **HotpotQA fullwiki train split**, fixing the well-known ground-truth issues that plague the original release (wrong question word, partial / single-form answers, presupposition mismatch, unrepairable rows). Each line is a JSON object that supersedes the original GT for a single HotpotQA `id`, with an explicit `verdict` that downstream pipelines filter by. Used as a **label-fix overlay** in the Twinkle RL stack to clean rewards for [`cookbook/rl/grpo_condensed.py`](https://github.com/modelscope/twinkle/blob/main/cookbook/rl/grpo_condensed.py) and as the audited GT source for the cold-start SFT builder ([`cookbook/rl/make_condensed_sft.py`](https://github.com/modelscope/twinkle/blob/main/cookbook/rl/make_condensed_sft.py)). ## Why re-annotate Auditing original HotpotQA against its own supporting passages reveals four recurring failure modes: - **GT type mismatch** — question asks "where" but GT is a person's name. - **Partial / incomplete** — multi-hop GT collapses to one of the two facts. - **Single surface form** — `"2"` only, when both `"2"` and `"two"` are valid. - **Malformed question** — wrong question word, truncated, or presupposition-mismatched against the answer type. These all silently corrupt F1-based RL rewards. This dataset records a strict-JSON verdict per row so callers can either keep / repair / drop each example without re-running the audit. ## What is in the dataset Each line is a JSON object. Core fields: | Field | Type | Description | | --- | --- | --- | | `id` | str | Original HotpotQA row id (matches upstream). | | `verdict` | str | One of `keep` / `fix_answer` / `fix_question` / `drop`. | | `question` | str | Original HotpotQA question (verbatim). | | `question_fixed` | str \| null | Repaired question when `verdict == 'fix_question'`; `null` otherwise. | | `original_answer` | str | Original single-form GT, kept for trace/debug. | | `answers` | list[str] | Multi-form gold answer list (empty for `drop`). | | `reasoning` | str | One-sentence audit rationale. | | `level`, `type` | str | Original difficulty / question type. | | `supporting_facts` | dict | Original HotpotQA supporting-facts structure (titles + sentence ids). | | `context` | dict | Original HotpotQA context (titles + sentences). | ### Verdict semantics | Verdict | Meaning | `answers` | `question_fixed` | | --- | --- | --- | --- | | `keep` | Original Q + A both correct. | multi-form expansion of original A | `null` | | `fix_answer` | Q is fine; A was wrong / incomplete. | corrected multi-form list | `null` | | `fix_question` | Q is malformed but repairable into a Q the SAME passages answer with the SAME gold facts. | multi-form list | repaired Q | | `drop` | Q cannot be repaired without changing the fact, OR passages don't support any answer. | `[]` | `null` | ### Multi-form answer rules The auditor expands each accepted GT to all valid surface forms before emitting `answers`: - **Number variants** — `"3"`, `"three"`, `"three-door"`, `"3-door"`. - **Range variants** — `"1901"`, `"1902"`, `"1901-1902"`, `"1901-2"`. - **Location variants** — `"Everett"`, `"Washington"`, `"WA"`, `"United States"`. - **Person variants** — legal name / nickname / full name. - **Entity-role pairs** — both the role and the entity for role-of-X questions. - **Show-vs-character pairs** — both the show and the character for best-known-for questions. - **Abbreviations** — `"NYC"` / `"New York City"` / `"New York"`. - **Title variants** — `"Dr. Smith"` / `"Smith"`. - **Date formats** — `"July 4, 1776"` / `"4 July 1776"`. Every answer is **short** (a name, entity, number, date, or yes/no) and **grounded** in the supporting passages — no hallucinated forms. ## How it was built Generated by [`cookbook/rl/reannotate_groundtruth.py`](https://github.com/modelscope/twinkle/blob/main/cookbook/rl/reannotate_groundtruth.py) against `hotpotqa/hotpot_qa:fullwiki:train`: 1. **Sampling** — stratified per-level sampling (`easy` / `medium` / `hard`) to the requested totals, with a `wrong_ids.txt` allow-list of known-bad rows force-included. 2. **Audit prompt** — each row is sent to a super-LLM (`qwen-max` or equivalent) together with its question, full context passages, and original GT. The system prompt locks the auditor to the four-verdict taxonomy and the multi-form / question-rewrite rules above. 3. **Strict-JSON parsing** — the response is parsed as a single JSON object; markdown fences are stripped, and a regex fallback recovers loose responses. Up to 3 retries per row on parse / API failure. 4. **Schema validation** — `verdict` must be in `{keep, fix_answer, fix_question, drop}`; `drop` rows must carry empty `answers` and `null` `question_fixed`; `fix_question` rows must carry a non-empty `question_fixed` distinct from the original. 5. **Resume-friendly write** — output is appended to a single JSONL; on re-run, already-written ids are skipped automatically. The result is a **12K-row overlay** stratified across HotpotQA levels, each row stamped with a verdict and a multi-form gold list grounded in its own passages. ## Usage ### As a label-fix overlay (RL training) ```python import json overrides: dict = {} drop_ids: set = set() with open('hotpotqa_fullwiki_reannotated_12k.jsonl', 'r', encoding='utf-8') as fh: for line in fh: obj = json.loads(line) rid = obj['id'] if obj['verdict'] == 'drop': drop_ids.add(rid) else: overrides[rid] = obj # Filter dropped rows; overlay fixed question / multi-form answers on the rest. def apply_overlay(row): if row['id'] in drop_ids: return None ov = overrides.get(row['id']) if ov is None: return {**row, 'answers': [(row.get('answer') or '').strip()]} return { **row, 'question': (ov.get('question_fixed') or row['question']), 'answers': ov['answers'], } ``` ### As an audited GT source (SFT data builder) The cold-start SFT builder [`cookbook/rl/make_condensed_sft.py`](https://github.com/modelscope/twinkle/blob/main/cookbook/rl/make_condensed_sft.py) consumes this overlay directly: `drop` rows are skipped, `fix_*` rows have their question / answers replaced before condensation and oracle rollout. ### Download ```python from modelscope import MsDataset ds = MsDataset.load('twinkle-kit/hotpotqa-full-wiki-refine-12k', split='train') ``` ## Reproducing ```bash python cookbook/rl/reannotate_groundtruth.py \ --output hotpotqa_fullwiki_reannotated_12k.jsonl \ --model qwen-max \ --api-key $OPENAI_API_KEY \ --base-url https://dashscope.aliyuncs.com/compatible-mode/v1 \ --total 12000 --easy 2000 --medium 4000 --hard 6000 \ --concurrency 16 --seed 42 ``` For the smaller "fix the known-bad cases only" run: ```bash python cookbook/rl/reannotate_groundtruth.py \ --output hotpotqa_reannotated_wrong.jsonl \ --model qwen-max \ --only-forced --wrong-ids cookbook/rl/wrong_ids.txt \ --concurrency 16 ``` ## License Apache License 2.0. The underlying HotpotQA passages and original questions remain under their original CC BY-SA 4.0 license; this dataset only adds an audit overlay (verdict + multi-form answers + optional repaired question).