Qwen3.8-27B-GGUF-metrics

# Qwen3.8-27B GGUF, everything behind the numbers This is the working record for [AtomicChat/Qwen3.8-27B-GGUF](https://huggingface.co/AtomicChat/Qwen3.8-27B-GGUF). Every figure in that model card came from a file in here, including the ones about other publishers' builds. The point of publishing it is simple. A quantization comparison is only worth reading if someone else can run it, and that needs three things nobody usually ships: the exact reference the numbers were measured against, the exact text they were measured on, and the exact rules each file was built with. All three are below. ## What is here | Path | What it is | |---|---| | `kld/` | the reference logits, split into parts, plus how to reassemble them | | `logs/base-neutral.log` | the run that produced the reference | | `logs/kld-neutral--*.log` | one measurement per file, ours and other publishers' | | `logs/quantize-*.log` | the full command and per-tensor result for each build | | `logs/imatrix-shard-*.log` | the importance matrix, one log per worker | | `logs/imatrix-merge.log` | merging those workers into one matrix | | `logs/imatrix-stats.log` | activation energy per tensor, sorted | | `logs/env.txt`, `logs/llama-commit.txt` | toolkit, driver, and llama.cpp commit | | `imatrix/imatrix.gguf` | the merged importance matrix, and its shards | | `results.json` | every measurement in one file, for plotting | ## How the measurements were made | | | |---|---| | Reference | the original BF16 weights, converted to GGUF, run unquantized | | Held-out text | `eval_neutral` from [AtomicChat/calib-corpora](https://huggingface.co/datasets/AtomicChat/calib-corpora), never used for calibration | | Context | 4096 | | Chunks | 87 | | Reference perplexity | 4.5219 plus or minus 0.0238 | | Metric | per-token KL divergence against the reference, plus top-1 agreement | | Hardware | 4x RTX 5090 | | CUDA | 13.0 | | llama.cpp | see `logs/llama-commit.txt` | The reference needs all four cards: the BF16 file is 51 GB. That is a property of the reference run only. The quantized files each fit on far less. > [!IMPORTANT] > Context is not a detail. The same file measured at 512 context and at 4096 > produces different divergence. Any number compared against ours has to use > 4096, the same held-out text, and the same reference, or it is measuring > something else. ## Using the reference yourself The `.kld` file holds the unquantized model's predictions over the whole vocabulary for every scored token. With it you can measure your own build against exactly the point we measured against, and put your number in the same table as ours. It is larger than the 50 GB single file limit, so it is stored in parts. The glob sorts correctly, so reassembly is one command: ```bash cat base-neutral.kld.*.part > base-neutral.kld stat -c %s base-neutral.kld cat base-neutral.manifest.txt ``` Then: ```bash llama-perplexity -m your-quant.gguf -f eval_neutral.txt \ --kl-divergence-base base-neutral.kld --kl-divergence -c 4096 -ngl 99 ``` There is deliberately no checksum. The hub already verifies transport, and the file is not bit reproducible across different GPU counts or drivers, so a checksum could not tell you whether you rebuilt the same reference. The byte count in the manifest is there to catch a missing part, which is the only failure the reassembly can actually have. ## The importance matrix Collected on the BF16 weights rather than on a quantized copy, over the `qwen3.8-27b` build in [AtomicChat/calib-corpora](https://huggingface.co/datasets/AtomicChat/calib-corpora): 4,967,044 tokens across 3,004 documents, rendered through the model's own chat template. | | | |---|---| | Source | BF16 | | Context | 512 | | Batch | 4096 | | Special tokens | parsed, not treated as punctuation | | Workers | 7, split by chunk range across 3 machines | Splitting works because the statistic is a sum over chunks. Each worker took its own range of the same corpus, so the union is the chunking a single run would have produced, and merging is addition rather than approximation: ```bash llama-imatrix -m Qwen3.8-27B-bf16.gguf -f calib_train.txt -o shard-0.gguf \ -ngl 99 -c 512 -b 4096 -ub 4096 --parse-special \ --from-chunk 0 --chunks 1400 llama-imatrix -m Qwen3.8-27B-bf16.gguf \ --in-file shard-0.gguf,shard-1.gguf,shard-2.gguf,... \ -o imatrix.gguf ``` `logs/imatrix-stats.log` has the activation energy per tensor, sorted. It is worth reading on its own: the highest values in the entire model sit on the attention gate of layers 52 to 62, with a second peak on layer 0. That is the measurement the layouts below are built on. > [!NOTE] > `--parse-special` is not optional here. Without it the chat markup in the > corpus is tokenized as ordinary punctuation, and the agentic and reasoning > parts calibrate on text the model never sees. ## What each file was built with Every build shares the same skeleton. Blocks 0 to 3 and 52 to 63 get one step above the base type, blocks 4 to 11 get the base type, `ffn_down` sits a step above `ffn_gate` and `ffn_up`, the attention gate and the state output sit a step above as well, `attn_k` and `attn_v` stay at `q8_0` because they are 0.3% of the weights each, and the multi token prediction head is pinned because it collects no calibration data. | File | `ffn_down` | `ffn_gate` / `ffn_up` | `attn_gate` | `ssm_out` | `output` | `token_embd` | |---|---|---|---|---|---|---| | `AD-Q6_K` | q6_k | q6_k | q8_0 | q8_0 | q8_0 | q8_0 | | `AD-Q6_K-Q5_K` | q6_k | q5_k | q8_0 | q8_0 | q8_0 | q5_k | | `AD-Q5_K` | q5_k | q5_k | q6_k | q6_k | q6_k | q4_k | | `AD-Q5_K-Q4_K` | q5_k | q4_k | q6_k | q6_k | q6_k | iq4_xs | | `AD-Q4_K` | q4_k | q4_k | q5_k | q5_k | q6_k | iq4_xs | | `AD-IQ4_XS` | q4_k | iq4_xs | q5_k | q5_k | q6_k | iq4_xs | | `AD-IQ4_XS-IQ3_S` | iq4_xs | iq3_s | q4_k | q4_k | q5_k | iq4_xs | | `AD-IQ3_S` | iq3_s | iq3_s | iq4_xs | iq4_xs | q5_k | iq4_xs | | `AD-IQ3_S-IQ3_XXS` | iq3_s | iq3_xxs | iq4_xs | iq4_xs | q5_k | iq4_xs | | `AD-IQ3_XXS` | iq3_xxs | iq3_xxs | iq3_s | iq3_s | q5_k | iq4_xs | | `AD-IQ2_S` | iq2_s | iq2_s | iq3_s | iq3_s | q5_k | iq4_xs | | `AD-IQ2_S-IQ2_XS` | iq2_s | iq2_xs | iq3_xxs | iq3_xxs | q4_k | iq4_xs | | `AD-IQ2_XS` | iq2_xs | iq2_xs | iq3_xxs | iq3_xxs | q4_k | iq4_xs | | `AD-IQ2_XXS` | iq2_xxs | iq2_xxs | iq2_s | iq2_s | iq4_xs | iq4_xs | | `AD-IQ1_M` | iq2_xxs | iq1_m | iq2_s | iq2_s | iq4_xs | iq4_xs | The exact command for any file, including the block ranges, is at the top of its `logs/quantize-<name>.log`. ## Why the layouts look like this Ten versions of the same file, same size class, changing only where the extra bits went. All ten were measured against the same reference on the same text. | Layout | Size | KL divergence | |---|---:|---:| | every layer treated the same | 16.8 GB | 0.01580 | | 4 layers lifted | 17.1 GB | 0.01449 | | more bits on `ffn_down` everywhere | 17.8 GB | 0.01189 | | more bits on attention | 18.2 GB | 0.01010 | | 16 layers lifted, first and last | 17.8 GB | 0.00981 | | 32 layers lifted instead | 18.4 GB | 0.00826 | | 16 lifted, plus the attention gate | 18.4 GB | 0.00821 | | 16 lifted, plus a richer output head | 18.8 GB | 0.00800 | | 24 layers lifted | 18.6 GB | 0.00743 | | 24 lifted, plus attention gate and state output | 18.6 GB | 0.00730 | Half the divergence disappears at the same size, from placement alone. Three things that fell out of it: The ends of the network are worth more than the middle, and the tail more than the head. Widening the lifted band past 24 layers stopped paying. This model is a hybrid. The attention gate and the state output are 5.5% of the weights each, and one extra step on both cost 0.16 GB and removed 11% of what was left. That was the best single trade we found. The embedding table is cheaper than its 4.7% suggests, and the output head is dearer. Paying for the head out of the embedding table won at every size. These files are not in the model repo. They were experiments, not releases, but their logs are here under the names `AD-Q4_K-L0`, `-E8`, `-E12`, `-W16`, `-G1`, `-G2`, `-H8`, `-A6`, `-D5`. ## Other publishers' files Their published figures are measured against their own reference and harness, so they cannot go in a table with ours. We downloaded their files and measured them here instead, against the same reference, on the same text, with the same build of llama.cpp. Those logs are named with the publisher first, for example `kld-neutral--unsloth--Qwen3.8-27B-Q4_K_M.log`. We are not claiming their numbers are wrong. We are claiming that ours and theirs can only be compared if they were produced the same way, and this is the only way we could make that true. ## Notes worth keeping `Q8_0` is not lossless on this model: 0.00064 divergence, 98.92% top-1. Every number here is measured against real BF16 rather than against a `Q8_0` stand in, which makes all of them larger than they would otherwise look. The multi token prediction head is never executed in a normal forward pass, so the importance matrix has nothing on it at any corpus size. Quantize it low and llama.cpp stops partway through rather than guess. Every tensor row in this model divides by 256, so the entire K and I quant range is genuinely available. That is not true of every recent release: when rows do not divide, llama.cpp silently substitutes a coarser type and keeps the name that was asked for.