NN6: The S-Invariant Attractor for Transformers: Convergence, Diagnostics, and Pruning

The S-invariant attractor provides a principled convergence criterion for feedforward networks: training is complete when each weight matrix becomes antisymmetric (W = -W^\top). But transformers are not feedforward. Their core operation—attention—is a dynamic interaction between queries, keys, and values that has no direct analogue in the weight-space decomposition of multilayer perceptrons. This paper extends the S-invariant convergence criterion to transformer architectures. What this paper provides: · Symmetry operators for each component of attention. For the query, key, and value projections: \mathcal{S}[W_Q] = -W_K, \mathcal{S}[W_K] = -W_Q, \mathcal{S}[W_V] = -W_V^\top. These definitions satisfy \mathcal{S}^2 = I and reflect the complementary roles of queries and keys in the attention product QK^\top.· The S-invariant attractor for attention. The attention mechanism reaches equilibrium when W_Q = -W_K and W_V = -W_V^\top. The query and key projections become mutual negatives; the value projection becomes antisymmetric. At this attractor, the attention pattern becomes symmetric in queries and keys, encoding pure directional relationships between tokens.· Convergence measures for transformers: C_Q(W_Q, W_K) = \|W_Q + W_K\|_F / (\|W_Q\|_F + \|W_K\|_F), C_V(W_V) = \|W_V + W_V^\top\|_F / (2\|W_V\|_F), and C_{\text{FF}}(W) = \|W + W^\top\|_F / (2\|W\|_F). All measures range from 0 (fully converged) to 1 (fully baseline). For multi-head attention, these measures apply per head.· Empirical validation on WikiText-2 language modeling. The S-invariant stopping criterion (all layers C < 0.05) achieves validation perplexity 66.3 \pm 0.3, outperforming early stopping with patience=5 (68.2 \pm 0.8) and patience=10 (67.1 \pm 0.5), with the lowest variance across runs.· Layer-wise convergence patterns. The first two layers converge to S-invariance; layers 5–6 do not converge within the training budget. This quantifies the well-known phenomenon that deeper transformer layers are undertrained, providing a per-layer measurement without separate validation sets.· Per-head convergence rates. Heads within the same layer converge at dramatically different rates. Heads 1–3 in layer 1 converge within 12 epochs; heads 4–8 in the same layer do not converge at all within 50 epochs. This variation is invisible to the loss curve but clearly visible to C_Q.· S-invariant pruning of redundant attention heads. Heads that reach C_Q < 0.05 within the first 15 epochs have learned trivial attention patterns and can be pruned. On WikiText-2, pruning 5 heads (10.4% of all heads) reduces parameters by 15% (44M to 37M) with a negligible increase in perplexity (66.3 \to 66.8), outperforming both random pruning (69.1) and magnitude pruning (68.7).· Diagnostic applications: Identifying undertrained layers (high C_Q while validation loss plateaus), identifying redundant attention heads (rapid convergence to near-zero C_Q), architecture diagnostics (identifying the depth at which directional learning stops), and training efficiency (freezing converged heads early). Why this matters: The same law that governs the Riemann zeros governs your transformer. The same Feed-backwards (Steering) dynamics \cite{Emergence35} that draw the zeros toward the critical line drive attention weights toward W_Q = -W_K and W_V = -W_V^\top. The practitioner can monitor C_Q, C_V, and C_{\text{FF}} alongside validation loss and stop training when all measures fall below 0.05. No patience parameter. No guesswork. Just a number, derived from first principles, that tells you when your transformer is done. Keywords: S-invariant attractor, transformer convergence, attention mechanism, query-key symmetry, value antisymmetry, convergence measures, attention head pruning, language modeling, WikiText-2, Canvas Model, Steering dynamics, Riemann zeros