whittle-stop-kd

# Whittle stop-KD > ### ☕ Support this work > Whittle is built by one person on a grocery budget and rented GPU hours, and this dataset was generated and released from that budget. > If it is useful to you, or you want to see the work continue: **[ko-fi.com/davida81328](https://ko-fi.com/davida81328)**. Every hour of GPU time goes into > the next release, and every trace, table and log lands in these repos. Top-32 teacher logits over 164 verified-complete answers, captured for training a model to know **when to stop** — not to be shorter, but to end where its teacher ends. Produced for the [Whittle](https://huggingface.co/logic65) project, which carves Qwen3.8-27B into a post-hoc mixture-of-experts. Released in case the capture recipe or the tail-weighting scheme is useful to anyone else working on repetition and failure-to-terminate. **Read before using (found 27 Aug 2026).** `kd_mt_top32.npz` is misaligned and must not be used, and `kd_top32.npz` is correctly aligned but its tail weighting never reaches the terminator. The details and numbers are under *Contents* and *Tail weighting* below; `capture_kd_mt_fixed.py` in this repo has both fixes. ## The problem this addresses Repetition in a compressed model is usually not a repetition problem, it is a failure to stop. Measured on the carved model against 36 held-out counted-list prompts, the base model emitted EOS on **15 of 36** and overran the requested item count by a mean factor of **2.85x** — asked for 10 items it produced 40, asked for 35 it produced 216. The answers were not degenerate; they simply never ended. Training on sampled text gives one hard label per position. Teacher logits give the full distribution, including how confident the teacher was that the answer should end *here* — which is the quantity actually being learned. ## Contents | file | what it is | |---|---| | `kd_top32.npz` | the capture: top-32 teacher distributions, labels, per-position loss weights, row offsets | | `kd_meta.json` | capture config, plus the teacher's `P(EOS)` over the last 24 positions of every answer | | `train_164_answers.jsonl` | the source answers the logits were captured over | | `capture_kd.py` | the script that produced it | **`kd_mt_top32.npz`, also in this repo, is misaligned and must not be used (found 27 Aug 2026).** Its per-turn spans were computed against a throwaway per-turn sequence and then stored against the full conversation, so only **43 of 258** weight-8.0 positions land on the turn terminator; the other 215 land on token 236, a partial UTF-8 byte. Corrected captures were to replace the `.npz` files here (27 Aug 2026). At the time of this card revision they have not: `kd_top32.npz` (13.84 MiB) and `kd_mt_top32.npz` (20.31 MiB) were both last changed on 27 Aug 2026, before the correction was written. The tree also holds `kd_v3.npz` and `kd_mt_meta.json`, which this card does not describe (see *Other files in this repo*). ### `kd_top32.npz` | array | shape | dtype | meaning | |---|---|---|---| | `idx` | (141465, 32) | int32 | token ids of the teacher's top-32 at each position | | `prob` | (141465, 32) | float16 | their probabilities (renormalise before use) | | `label` | (141465,) | int32 | the actual next token | | `weight` | (141465,) | float32 | per-position loss weight (see below) | | `offsets` | (164, 2) | int64 | `(start, length)` into the above, one row per answer | **Top-32 captures 99.3–99.8% of the probability mass**, so truncating to 16 at training time loses very little. 32 was captured because storage is trivial and re-running the teacher is not. ## Tail weighting The stop decision lives in the last few positions of an answer. A uniform loss over a 500-token answer lets roughly 497 "keep writing" positions drown the three that say "end here". The weights are therefore: ``` body 0.25 last 8 positions ramp 1.0 -> 4.0 the position predicting EOS 8.0 ``` **The body weight is deliberately non-zero.** Zeroing it teaches "stop after N tokens" — a length prior — rather than "stop when the answer is complete". In an earlier run on this project a model that learned the length prior scored exact counts across its training range and then collapsed at the first count above it, producing 12 items when asked for 40. **What the shipped capture actually does (found 27 Aug 2026).** `kd_top32.npz` is correctly aligned, but its tail weighting does not do what the scheme above claims. The capture stops one position short of the terminator, so the position carrying weight 8.0 is the one *before* the answer ends. At that position the teacher does not want to stop - it predicts `\n` in **120 of 164** answers at **p = 0.746**, and the terminator in only 25. Weighting it at 8.0 against a body weight of 0.25 therefore upweights **continuing** by 32x. Measured consequence, on 36 held-out counted prompts: mean overrun rose with trainable capacity rather than falling - **2.85x untrained, 3.62x with a 0.33M gate, 6.39x with 147M of LoRA**, the last stopping on only 5 of 36 prompts. **If you are using this set for termination**, add explicit supervision on the terminator position itself; the KD arrays never reach it. `capture_kd_mt_fixed.py` in this repo has both fixes and asserts the alignment before saving. ## Source answers 164 single-turn answers generated by **Qwen/Qwen3.8-27B** (apache-2.0) over generic prompts — counted lists, open-form explanations, and structured output such as SQL, YAML and HTML. Every answer was verified before capture: - `finish_reason == "stop"` — the teacher ended it, nothing was truncated - windowed 4-gram repetition below 0.15 — the teacher did not loop - for counted prompts, at least 80% of the requested items were delivered - for code prompts, the block was properly fenced Answers the teacher itself looped on or left unfinished were discarded rather than captured. A teacher that fails to stop cannot teach stopping. **Windowed, not global, repetition.** Global 4-gram repetition averaged over a whole generation cannot see tail degeneration: a 30-item list that runs clean for 20 items and then falls apart scores **0.061** against a 0.15 threshold — a comfortable pass — because the clean head dilutes the ratio. The same text scores **0.982** when the metric takes the worst 60-token window. The dilution measures how long the healthy prefix is, not how bad the loop is. ## Using it ```python import numpy as np, json, torch z = np.load("kd_top32.npz") meta = json.load(open("kd_meta.json")) start, length = z["offsets"][i] # answer i idx = z["idx"][start:start+length] # (L, 32) prob = z["prob"][start:start+length] # (L, 32) w = z["weight"][start:start+length] # (L,) # forward KL to the teacher's top-k, per-position weighted tp = prob / prob.sum(-1, keepdims=True) logp = torch.log_softmax(student_logits, -1).gather(-1, idx) loss = ((-(tp * logp).sum(-1)) * w).sum() / w.sum() ``` Positions align to the **answer span only** — the prompt is excluded, and the final position is the one predicting EOS (as designed; in the shipped `kd_top32.npz` the capture stops one position short of the terminator — see the 27 Aug 2026 note under *Tail weighting*). `offsets` maps answers to slices; to reconstruct the token sequence, apply the chat template to `prompt`, tokenize `text`, and append EOS. ## Other files in this repo (not described on this card) Present in the tree, names only; this card does not describe their contents: `abort_eval.py`, `antiloop_prompts_v3.json`, `arith_prompts.json`, `build_prompts_v3.py`, `capture_kd_mt.py`, `colab_run.py`, `eval_arms.py`, `eval_fix1_multiturn.json`, `eval_results_27aug.json`, `filter_exact.py`, `fix1_gate_weights.pt`, `gate_sweep.py`, `grpo_count.py`, `harvest_cot.py`, `harvest_multiturn.py`, `kd_mt_meta.json`, `kd_v3.npz`, `mathprobe.py`, `merge_kd.py`, `prep_kd_cot.py`, `prep_kd_dataset.py`, `teacher500.jsonl`, `teacher_arith.jsonl.gz`, `teacher_arith_v2.jsonl.gz`, `teacher_cot.jsonl`, `teacher_cot_r2.jsonl.gz`, `teacher_multiturn.jsonl`, `train_stop_v2.py`, `train_v3.jsonl`. ## Provenance and licence Teacher: `Qwen/Qwen3.8-27B`, apache-2.0. This dataset consists of that model's outputs and distilled top-k distributions over them, released under the same licence. No personal data. The prompts are synthetic and generic; the answers contain only illustrative placeholders where they discuss addresses or contacts — for example `93.184.216.34`, the IANA reserved address for `example.com`, in a passage explaining DNS. ## Caveat These 164 answers are **single-turn only**. A gate trained on them alone improved counted-list stopping substantially and then regressed multi-turn conversation — mean windowed repetition roughly doubled, and tripled in later turns — because the trained component is active on every token of every turn but had never seen a dialogue. If you use this for a component with global reach, mix in multi-turn data and hold out multi-turn as a gate before shipping.