Decoding-Level Taboo: A Diagnostic Stress Test for LLM Robustness

Decoding-Level Taboo A Diagnostic Stress Test for LLM Robustness Anonymized code and result summaries for the paper "Decoding-Level Taboo: A Diagnostic Stress Test for LLM Robustness" (under review). Author-identifying information has been removed for double-blind review; the de-anonymized repository will be linked on acceptance. Decoding-Level Taboo is a zero-prompt diagnostic that masks a model's top-i candidate tokens at word boundaries during greedy decoding, forcing "machine circumlocution" and measuring how much of a model's multi-step reasoning survives being pushed off its preferred generation path. Layout taboo/ core: word-start detection, the taboo logits processor, surprisal metrics word_start.py vocabulary-level word-boundary mask (BPE "Ġ" / SentencePiece "▁") decoder.py TabooLogitsProcessor (rank-i mask at word starts) + injected surprisal ΔS metrics.py conditional / marginal retention, recovery cluster/ taboo_sweep_batched.py main HuggingFace runner (batched sweep over model/task/condition; --base_prompt few-shot, --load_4bit, --temperature/--seed, --save_gen) eval/ scoring + analysis (conditional retention, IFEval-by-constraint, temperature summary, failure-example extraction, LLM-judge MCQ re-scoring) paper/ figure scripts (fig_*.py) and result summaries in paper/data/ tests/ unit tests for the decoder core Reproducing python -m venv .venv && . .venv/bin/activate pip install torch transformers accelerate bitsandbytes datasets # one (model, task) sweep over the taboo dose, few-shot base_prompt, 4-bit: python cluster/taboo_sweep_batched.py \ --models Qwen/Qwen2.5-7B Qwen/Qwen2.5-7B-Instruct \ --tasks gsm8k --conditions baseline taboo_i1 taboo_i2 taboo_i4 taboo_i8 \ --base_prompt --load_4bit --n 100 --outdir results # conditional retention (per-item matched within a run): python eval/conditional_retention.py results/sweep_summary.jsonl # figures (from the shipped summaries in paper/data/): python paper/fig_heatmap.py The result summaries used in the paper are in paper/data/ (summaries_a*.jsonl, cond_ret_a*_clean.json, etc.), so the figures reproduce without re-running the sweeps. Notes Word-start masking is the decisive design choice: masking mid-word instead collapses accuracy to noise (tokenization corruption), while word-start masking applies semantic stress. The optional MCQ re-scoring in eval/gemini_parse.py reads an API key from the environment (GEMINI_API_KEY); it is used only as a sanity check and is not required to reproduce the main deterministic results.