Research map / Architectures and efficiency / RNNs and normalization
Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobians
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes recurrent self-attention dynamics without energy-function constraints via Jacobians, showing normalization suppresses Jacobian norm and oscillatory eigenvalues, that high-performing models sit near Lyapunov criticality, and proposes Jacobian spectral regularization for looped Sudoku solvers.
Problem
Energy-based theory of self-attention dynamics relies on idealized constraints (symmetric weights, single head, continuous time) that do not hold in standard SA, leaving inference dynamics of realistic discrete, normalized SA poorly understood.
Approach
Relaxes symmetry and single-head energy constraints to derive generalized energy functions, then uses Jacobian spectral analysis (spectral norm, complex eigenvalues, Lyapunov exponents) to characterize energy-agnostic dynamics, showing normalization stabilizes the dynamics and places high-accuracy models near criticality.
Key results
- Spectral regularization (Spec) substantially improves both AKOrN and ItrSA on Sudoku, with the largest gain for AKOrN, while energy-based regularizers (E-single, E-multi) underperform the unregularized baselines.
- High-accuracy models have maximum Lyapunov exponents near zero (about 0.1, slightly positive), while energy-based symmetric SA models show negative exponents and lower accuracy.
- ItrSA with RMSNorm exhibits test-time scaling (accuracy improves with more loop iterations), whereas AKOrN loses it when oscillator dimension N is large.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of loops T | ItrSA · Sudoku (SATNet ID and RRN OOD) | accuracy | Helped |
| Add Jacobian spectral regularization (Spec) | AKOrN · Sudoku | accuracy | Helped |
| Add Jacobian spectral regularization (Spec) | ItrSA · Sudoku | accuracy | Helped |
| Add energy-based multi-head regularization (E-multi) | ItrSA · Sudoku | accuracy | Hurt |
| Add energy-based single-head regularization (E-single) | ItrSA · Sudoku ID | accuracy | Hurt |
| Apply oscillator-wise RMS normalization | AKOrN · Sudoku test-time scaling | accuracy | Helped |
| Use normalization in the recurrent dynamics | Self-attention models · Sudoku | maximum Lyapunov exponent | Helped |
| Constrain dynamics with energy-based symmetric self-attention | Energy-based symmetric SA · Sudoku | accuracy | Hurt |
| Increase oscillator dimension N (ablation) | AKOrN · Sudoku test-time scaling | accuracy | Hurt |
| Increase the number of attention heads (ablation) | Self-attention models · CIFAR-10 and language modeling | Lyapunov exponent | Helped |
Surprising findings
- Energy-based regularization that enforces the energy-function constraints reduces performance rather than helping, casting doubt on the practical value of the energy perspective.
- Successful models have a slightly positive maximum Lyapunov exponent, suggesting chaotic-side criticality rather than convergence.
Limitations
- Analysis excludes positional encoding, masking, and MLP blocks present in practical looped Transformers.
- The theoretical spectral-norm bound is conservative compared with empirical observations.
- Experiments are limited to Sudoku and CIFAR-10 with a small number of seeds.
Experimental setup
Datasets: SATNet Sudoku RRN Sudoku CIFAR-10
Benchmarks: SATNet Sudoku RRN Sudoku CIFAR-10
Metrics: accuracy training loss maximum Lyapunov exponent spectral norm of Jacobian pseudo-energy
Baselines: AKOrN ItrSA (without oscillator operations) E-single E-multi
Builds on: AKOrN energy-based self-attention (particle dynamics) Lipschitz constant bound for self-attention spectral norm regularization
Closest papers by meaning
- High-Dimensional Learning Dynamics of Attention-Indexed Models (NeurIPS 2026 · similarity 0.87)
- Exponential Dynamic Energy Network for High Capacity Sequence Memory (NeurIPS 2025 · similarity 0.87)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.86)
- Identifying latent state transitions in non-linear dynamical systems (ICLR 2025 · similarity 0.86)
- Self-attention Networks Localize When QK-eigenspectrum Concentrates (ICML 2024 · similarity 0.86)
- Characterizing control between interacting subsystems with deep Jacobian estimation (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.