cds-jb/synthweb-qwen3.5-9b-multiscale-inference
收藏资源简介：
--- configs: - config_name: default data_files: - split: train path: data/train-* dataset_info: features: - name: doc dtype: string - name: scope dtype: string - name: verbalizer_prompt dtype: string - name: context dtype: string - name: atom_text dtype: string - name: target_response dtype: string - name: incorrect_plausible_answer dtype: string - name: bb_answer_rollout_answers list: string - name: bb_answer_rollout_scores list: float64 - name: bb_answer_score_mean dtype: float64 - name: bb_answer_score_max dtype: float64 - name: bb_answer_score_min dtype: float64 - name: row_seed dtype: int64 - name: doc_id dtype: string - name: doc_source dtype: string - name: doc_idx dtype: int64 - name: split_char_offset dtype: int64 - name: row_splits struct: - name: word_lens dtype: int64 - name: sentence dtype: int64 - name: paragraph dtype: int64 - name: whole dtype: int64 - name: doc_choice_rationale dtype: string - name: split_rationale dtype: string - name: slot_idx dtype: float64 - name: target dtype: string - name: typicality dtype: string - name: abs_index dtype: float64 - name: signed_index dtype: float64 - name: signed_index_hint dtype: float64 - name: actual_word_distance_from_split dtype: float64 - name: synergy_check dtype: string - name: distribution_check dtype: string - name: difficulty_check dtype: string - name: source dtype: string - name: n_tokens_actual dtype: float64 - name: tokenizer dtype: string - name: generator_model dtype: string - name: source_model dtype: string splits: - name: train num_bytes: 6150303735 num_examples: 590741 download_size: 979308451 dataset_size: 6150303735 --- # synthweb-qwen3.5-9b-multiscale-inference Probing-question dataset built over [`cds-jb/synthweb-qwen3.5-9b`](https://huggingface.co/datasets/cds-jb/synthweb-qwen3.5-9b) (Qwen3.5-9B) continuations of FineWeb prefixes. Each row is ONE probe testing whether content on one side of a character-level split in a prefix+continuation can be recovered from the **latent hidden state** of the source LM at that split. This dataset is the evaluation harness for **method M** — an "activation oracle" that decodes hidden-state content into natural language. ## Why this dataset Activation-oracle methods claim to recover information that is encoded in a language model's hidden state but NOT yet manifest in the surface text. To measure that capability we need probes that satisfy two constraints: 1. **HARD-FROM-TEXT** — a careful reader of the side OPPOSITE to the target should NOT be able to confidently produce the answer from the complement text alone (no verbatim quote, no clean paraphrase). 2. **EASY-FROM-LATENT** — the answer should be exactly the kind of content the source LM has already COMMITTED TO in its hidden state at the split (the next thematic move it was primed for; the specific named entity it has been tracking; the conclusion it was building toward). Probes that fail (1) measure nothing M adds beyond a competent text reader. Probes that fail (2) measure entropy, not oracle skill. The **`bb_answer_score_max`** column in this dataset is precisely the (1) test — a low value flags a good HARD-FROM-TEXT probe. For HARD-FROM-LATENT, each spw probe carries a `target_response` field — the answer a "Big Brother" (BB) monitor would give if it had access to **both** the complement context AND `atom_text` (i.e. the full-info ground truth, generated by Haiku in one joint call). The contrast between `target_response` (full-info answer) and the **blind** `bb_answer_*` columns (complement-only answer) operationalizes the gap M is supposed to close. ## Probe design — five scope groups Each source doc (FineWeb prefix + sibling Qwen3.5-9B continuations) yields up to **16 probes** spanning five scope groups: | scope | n in this build | atom (what's being recovered) | scoring path | |---|---:|---|---| | word | 184,959 | a single content word | local (exact-match / token-logprob) | | lens | 184,805 | contiguous N tokens past/future the split | local (Qwen tokenizer) | | sentence | 110,882 | one sentence | Haiku answerer + judge | | paragraph | 73,932 | one paragraph | Haiku answerer + judge | | whole | 35,622 | one half-doc (≥ all paragraphs on one side) | Haiku answerer + judge | **Total probes in this build: 590,741** across 36,997 distinct source docs (3 round(s)). Per-doc slot allocation is **5 word + 5 lens + 3 sentence + 2 paragraph + 1 whole = 16 slots/row**. Word and lens are Python-templated (no LLM); sentence / paragraph / whole are written by Claude Haiku 4.5 in extended-thinking mode (see "Generator" below). ### Per-scope splits Each row carries **four** char-level split offsets — one per scope group: - `word_lens` split → applies to the 10 word + lens probes - `sentence` split → applies to the 3 sentence probes - `paragraph` split → applies to the 2 paragraph probes - `whole` split → applies to the 1 whole probe Haiku picks all four split points in a single response. It may use the same offset for all four (≈51% of rows in the smoke), or up to four distinct offsets when different scopes need different boundaries (≈49% of rows). The per-probe `split_char_offset` field gives the offset that applies to THAT probe. ### Targets and typicality For each probe Haiku picks a `target ∈ {prefix, suffix}`: - **prefix-target** → M sees the suffix; must POST-dict an earlier-in-doc atom. - **suffix-target** → M sees the prefix; must PRE-dict a later-in-doc atom. For suffix-target slots, Haiku additionally labels `typicality ∈ {typical, atypical}` against the cross-sibling distribution: - `typical` → the specific claim/event/move recurs in at least 3 other sibling continuations at a comparable position. - `atypical` → the claim is essentially unique to the chosen sibling and absent from the rest. The `distribution_check` field records Haiku's justification. ### Synergy constraint (per scope) Each multi-token atom (sentence / paragraph / whole) must require **synthesizing information across the entire atom**, not a strict sub-region: > *sentence* → must integrate the main clause with its qualifying > clauses / numbers / objects. > *paragraph* → must combine at least two of the paragraph's sentences. > *whole* → must integrate across multiple paragraphs/sections (an > arc, a thesis-evidence chain, a tension between stated stances). The `synergy_check` field per probe names which sub-pieces must be integrated. ### Incorrect plausible answer Every probe (incl. word & lens) carries an `incorrect_plausible_answer` (IPA) — a hypothetical alternative answer a careful reader could PLAUSIBLY produce from the complement side, that happens to be wrong. This enables contrastive scoring (`logP(atom)` vs `logP(IPA)` under M). The IPA must match the atom's shape (single word for word; same-length sentence for sentence; etc.) and must NOT be what any sibling actually produced. ## Generator **Model**: Claude Haiku 4.5 (`claude-haiku-4-5-20251001`) in extended-thinking mode (8K thinking-budget, 28K max output tokens), submitted via Anthropic Message Batches API for ~50% cost reduction. **What Haiku sees per row**: the FineWeb prefix + up to 3 sibling Qwen3.5-9B continuations of that prefix (sampled from the source rollouts dataset). The siblings give Haiku a sense of the source LM's distribution at this point, which it uses to label typicality and to craft IPAs that don't accidentally match a real sibling. **Prompt scaffold** (lightly abridged from `scripts/oracle_question_prompt.py`): > You design probing questions for evaluating a method M. > > What M does: M is given (i) the PREFIX of a document and (ii) the > LATENT HIDDEN-STATE of the source LM at the boundary between prefix > and suffix — the same source LM that generated the suffix. M's > capability is to use that hidden state to read off content the source > LM was about to emit (for SUFFIX-target probes) or had committed to > earlier in the document (for PREFIX-target probes). > > The goal of each probe: construct a question whose answer is > HARD-FROM-TEXT but EASY-FROM-LATENT. > > [...] > > Self-check before finalizing each probe: mentally try to answer your > verbalizer using ONLY the complement-side text. If you can produce > atom_text (or a clean paraphrase) from that alone, the probe is too > easy on text — pick a different atom or rewrite the verbalizer to > require inference about what the source LM was INTERNALLY tracking, > not what is already surface-visible. **Calibration**: rounds were tuned on a 100-doc calibration sweep with addendum levels {-3..+3} sweeping HARD-FROM-TEXT strictness; the build settled on **level -2** ("probes have been too hard. Pick atoms that share an obvious topical / narrative thread with the complement so a text-only reader has solid footing, while still leaving the SPECIFIC content of the atom for M to recover"). This produced a mean answerer-max score of ~0.50 with substantial bimodality (see below). **JSON robustness**: ~1.3% of Haiku responses had small JSON quirks (unescaped quotes inside long string fields, raw newlines). These are recovered with [`json-repair`](https://pypi.org/project/json-repair/), lifting end-to-end finalize rate from ~86% (strict parsing only) to 98.7%. ## Answerer + judge (scoring pass) For sentence / paragraph / whole probes, this dataset additionally carries 5 stochastic answerer rollouts (Haiku 4.5, temperature=1.0, max_tokens=800) and a per-rollout judge score from another Haiku 4.5 call (0.0–1.0). The answerer sees ONLY the complement side of the split (up to 4000 chars) plus the verbalizer, and is asked to produce a concise answer. The judge sees the verbalizer, the ground-truth `atom_text`, and the model answer, and emits a single float score: | score | meaning | |---|---| | 0.0 | totally wrong / unrelated | | 0.5 | partially correct or correct theme but wrong specifics | | 1.0 | essentially the same content as ground truth | The **max** across 5 rollouts is the headline filter signal: if even the best of 5 stochastic attempts can't recover the atom from the complement, the probe is HARD-FROM-TEXT — exactly what you want to keep for evaluating M. ### Max-score histogram (this build) ``` [0.0, 0.2) 33,518 ( 17.7%) ######## HARD [0.2, 0.4) 45,525 ( 24.1%) ############ [0.4, 0.6) 25,803 ( 13.7%) ###### valley [0.6, 0.8) 52,087 ( 27.6%) ############# [0.8, 1.0] 31,961 ( 16.9%) ######## EASY ``` mean=0.461 std=0.296 · HARD (max<0.4) 41.8% · EASY (max≥0.8) 16.9% The clear central valley around [0.4, 0.6) is the filtering signal: keep the HARD tail for M-evaluation; the EASY tail is text-derivable and serves as a sanity check. ### Cost & methodology of the answerer pass - spw probes scored: **188,819** - answerer calls: 5 × 188,819 = 944,095 - judge calls: 5 × 188,819 = 944,095 - estimated tokens: ~1227.3M input + ~75.5M output (answerer) + ~377.6M input + ~4720.0K output (judge) - batch-discounted Haiku 4.5 ($0.50/M input · $2.50/M output): **~$1003** - wall time: ~3–4 hours on Anthropic's batch queue (2 answerer chunks ≤256 MB each, plus 2 judge chunks ≤100,000 requests each — the Anthropic Message Batches API has a per-batch hard cap of 100K requests). If you want to re-score with more rollouts (denoising the per-probe max-score estimate), reuse `scripts/score_probes.py --model qwen3.5-9b --n-rollouts K` — the cost is linear in K. ## Schema (per row) Each row is one probe. Key fields: | column | type | meaning | |---|---|---| | `doc_id` | str | FineWeb-derived id | | `doc_source` | str | original FineWeb URL | | `doc` | str | the chosen sibling continuation (prefix + suffix) | | `doc_idx` | int | which sibling Haiku committed to | | `split_char_offset` | int | character split for this probe's scope group | | `row_splits` | dict | all four scope-group splits for the row | | `slot_idx` | int | 0..15 within the row | | `scope` | str | word / lens / sentence / paragraph / whole | | `target` | str | prefix or suffix | | `typicality` | str/null | typical / atypical (suffix only) | | `abs_index` | int | absolute atom index | | `signed_index` | int | signed atom index (negative = prefix, positive = suffix) | | `atom_text` | str/null | text of the target atom | | `verbalizer_prompt` | str/null | the probing question | | `incorrect_plausible_answer` | str/null | one hypothetical wrong-but-plausible alternative | | `synergy_check` | str/null | which sub-pieces must be integrated (multi-token scopes) | | `distribution_check` | str/null | relation to cross-sibling distribution (suffix slots) | | `difficulty_check` | str/null | the per-probe HARD-FROM-TEXT / EASY-FROM-LATENT argument | | `source` | str | python_template or haiku | | `n_tokens_actual` | int/null | actual token count for lens probes | | `tokenizer` | str/null | tokenizer used (Qwen3-8B) for lens probes | | `generator_model` | str | always `claude-haiku-4-5-20251001` | | `source_model` | str | `qwen3.5-9b` | | `bb_answer_rollout_answers` | list[str]/null | 5 model answers (spw probes only) | | `bb_answer_rollout_scores` | list[float]/null | 5 judge scores | | `bb_answer_score_mean` | float/null | mean of the 5 scores | | `bb_answer_score_max` | float/null | **max — primary filter signal** | | `bb_answer_score_min` | float/null | min | | `target_response` | str | full-info correct answer (Haiku sees context + atom_text); for word/lens probes this equals `atom_text` | | `context` | str | complement-side raw text (the side opposite to atom_text relative to the scope's split) | ## Reproducibility & incremental growth The dataset is built in **rounds**. Each round samples N **new** doc-ids — disjoint from prior rounds — using a deterministic seed. The local manifest at `data_pipelines/multiscale_inference/qwen3.5-9b/manifest.json` records `(round_idx, batch_ids, doc_ids, seeds, submitted_at)` per round, so a follow-up extension run will pick up where this one left off without overlap. Scripts (in [github repo TBD]): - `scripts/submit_multiscale_inference.py` — submit a round of generator batches - `scripts/poll_multiscale_inference.py` — fetch + finalize + push probes - `scripts/score_probes.py` — score with answerer + judge - `scripts/oracle_question_prompt.py` — the procedural prompt builder ## Source rollouts - [`cds-jb/synthweb-qwen3.5-9b`](https://huggingface.co/datasets/cds-jb/synthweb-qwen3.5-9b) — Qwen3.5-9B continuations of FineWeb prefixes (mode-collapse-filtered and detached-tail-truncated). - See that dataset's README for the rollout filter pipeline. ## License Same as upstream FineWeb (ODC-By 1.0) for the prefix text; generated continuations and probes are released under the same license for research use.
The dataset named synthweb-qwen3.5-9b-multiscale-inference is a probing-question dataset built over Qwen3.5-9B continuations of FineWeb prefixes. It is designed to evaluate a method called activation oracle (Method M), which decodes hidden-state content from a language model into natural language. Each row in the dataset represents one probe that tests whether content on one side of a character-level split in a prefix+continuation can be recovered from the latent hidden state of the source language model at that split. Probes must satisfy two constraints: 1) HARD-FROM-TEXT, meaning a careful reader of the opposite side cannot confidently produce the answer from the complement text alone; and 2) EASY-FROM-LATENT, meaning the answer should be content the source model has already committed to in its hidden state. The dataset contains 590,741 probes across five scope groups: word (184,959), lens (184,805), sentence (110,882), paragraph (73,932), and whole (35,622), with up to 16 probes per source document. It includes detailed fields such as document ID, split offsets, target response, incorrect plausible answer, synergy checks, and scoring results from answerer and judge models (Claude Haiku 4.5), with bb_answer_score_max as the primary filter signal. The dataset supports research on information encoding in language model hidden states and facilitates evaluation and reproducibility of activation-oracle methods.
