cds-jb/synthweb-qwen3-8b
收藏资源简介：
# qwen3-8b-fineweb-rollouts-100k Qwen3-8B continuations of FineWeb prefixes — 100,000 source docs from `HuggingFaceFW/fineweb` (`sample-10BT`), with up to 15 rollouts per prefix. Mode-collapse filtered AND **truncated at detached-end markers** (CoT scaffold / comprehension Q&A / summary coda — patterns that indicate Qwen3-8B has switched out of "continuing FineWeb prose" into post-training scaffold modes). ## Filtering pipeline 1. **Pool**: 98,827 FineWeb docs filtered to match the CoT-oracle length distribution (`data_pipelines/fineweb_oracle/build_doc_pool.py`). 2. **Generation**: Qwen3-8B (BF16, vLLM 0.20.0) generates 15 rollouts per doc with `temperature=1.0, top_p=0.95, top_k=20, max_new_tokens=768`. 3. **Mode-collapse filter** drops rollouts triggering any of: 12-gram word repetition ≥3×, CJK leakage, `<think>` tag, assistant boilerplate openers, alien markdown structure. 4. **Detached-end truncation**: scan the kept continuation for the first match of any post-training-scaffold marker (e.g. "Okay, let me", "Based on the article", "Question:", numbered MCQ, summary coda) and TRUNCATE `generated_text` at that point. The `tail_truncated_at_marker` column records which pattern matched (null if no truncation). 5. **Length floor**: rollouts whose truncated continuation has fewer than 400 characters are dropped (too short to be useful for activation probing). ## Schema (one row per surviving rollout) Same as the source parquet, plus: - `tail_truncated_at_marker` (str|null): regex pattern that triggered the truncation, or null if the continuation was kept whole. `generated_text` is the (possibly truncated) continuation. Use this column for activation-oracle probe construction. ## Why truncate rather than drop Many Qwen3-8B continuations are perfectly fine FineWeb-style prose for the first ~200-400 tokens and only later drift into a comprehension-Q&A or chain-of-thought tail (a fingerprint of Qwen3-8B's post-training data). Dropping the whole rollout loses useful prose; truncating at the boundary keeps the clean portion.
The dataset named qwen3-8b-fineweb-rollouts-100k is a collection of Qwen3-8B continuations of FineWeb prefixes. It includes 100,000 source documents from the sample-10BT subset of HuggingFaceFW/fineweb, with up to 15 rollouts per prefix. The dataset is filtered for mode-collapse and truncated at detached-end markers to remove parts where Qwen3-8B switches from continuing FineWeb prose into post-training scaffold modes (e.g., chain-of-thought scaffold, comprehension Q&A, or summary coda). The processing pipeline involves: document pool filtering to match length distribution, model generation, mode-collapse filtering (e.g., n-gram repetition detection), truncation at post-training markers (cutting generated_text at detected patterns), and a length floor (dropping rollouts with fewer than 400 characters). The schema matches the source parquet with an added tail_truncated_at_marker column to record the regex pattern that triggered truncation. Truncation is preferred over dropping entire rollouts to preserve useful prose portions before drift occurs.
