Research map / Architectures and efficiency / RNNs and normalization

Dynamics of the Transformer Residual Stream: Coupling Spectral Geometry to Network Topology

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Layer-wise analysis of language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Computes exact per-layer Jacobians of Llama 3.1 8B, OLMo 3 7B (across training checkpoints) and Gemma 4 E4B, characterizing their spectral geometry, cumulative low-rank bottleneck, and coupling between activation-graph community boundaries and Jacobian column norms.

Problem

How computation propagates through LLM layers is poorly understood; prior analyses of residual-stream dynamics used scalar summaries or approximate linearizations, and the relationship between residual-stream community topology and Jacobian dynamics is untested at production scale.

Approach

Uses exact input-space Jacobians with full eigendecomposition and Schur surgery across depth and training checkpoints (with a step-0 random-init null model) to separate architectural from learned properties, and links per-unit participation in signed activation-correlation communities to Jacobian column-norm amplification.

Key results

• About 98% of Jacobian eigenvalues occur as complex conjugate pairs; self-alignment rises from about 0.04 (layer 2) to about 0.70 (layer 29) in Llama 3.1 8B.
• Cumulative effective rank of the end-to-end Jacobian product falls to 6.7 in Llama 3.1 8B from 4,096 input directions; Schur surgery at c=0 raises it to 45.4 (6.4x).
• Boundary-node coupling is 0/32 FDR-significant layers at OLMo step 0, 14/32 at step 471k, and 24/32 at step 1.41M; Gemma shows 13/42 negative-coupling layers.

Surprising findings

• Control A (random feedforward with matched Frobenius mass) raises cumulative effective rank instead of lowering it, reversing the dose response.
• Gemma 4 E4B shows significant negative boundary-node coupling in strongly non-normal early and mid layers, reversing the sign seen in OLMo and Llama.

Limitations

• Jacobians evaluated only on last-token activations from WikiText-2 with 1,000 samples, so distribution shift is untested.
• Per-unit effect sizes are moderate; OLMo has only three checkpoints, so prolonged-training effects are unresolved.
• Community-detection results depend on detector choice and resolution parameter (CPM gamma=0.001).

Experimental setup

Datasets: WikiText-2

Benchmarks: WikiText-2

Metrics: eigenvalue distribution self-alignment Henrici departure effective rank condition number Cohen's d NMI variance captured by community-projected operator

Baselines: OLMo 3 7B step 0 (random initialization) Control A (Frobenius-matched random replacement) Control B (Haar-random Schur basis) |corr| InfoMap signed Leiden gamma=1 signed Leiden gamma=2

Builds on: signed Leiden CPM Schur decomposition dynamical isometry global workspace theory

Closest papers by meaning

• Transformer Block Coupling and its Correlation with Generalization in LLMs (ICLR 2025 · similarity 0.88)
• Transformers Represent Belief State Geometry in their Residual Stream (NeurIPS 2024 · similarity 0.87)
• HoloNets: Spectral Convolutions do extend to Directed Graphs (ICLR 2024 · similarity 0.87)
• Do Language Models Use Their Depth Efficiently? (NeurIPS 2025 · similarity 0.87)
• GIST: Gauge-Invariant Spectral Transformers for Scalable Graph Neural Operators (NeurIPS 2026 · similarity 0.87)
• Transformers need glasses! Information over-squashing in language tasks (NeurIPS 2024 · similarity 0.87)