Research map / Architectures and efficiency / Attention and transformers

Stability and Generalization in Looped Transformers

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Transformer expressivity and learnability · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a fixed-point framework (reachability, input-dependence, geometry) for looped transformers, proves autonomous looped nets have countable fixed points and recall plus outer normalization yields stable input-dependent fixed points, and introduces internal recall, evaluated on chess, sudoku, and prefix-sums.

Problem

Looped transformers promise test-time compute scaling, but it is unclear which architectural choices (recall, recall placement, outer normalization) let them extrapolate to harder problems rather than memorize training-depth solutions.

Approach

A fixed-point analysis along three stability axes (reachability, input-dependence, geometry) that theoretically explains why recall is needed and why outer normalization helps in looped models; plus a novel internal recall placement that is competitive with standard (external) recall once outer normalization is applied.

Key results

• Normalized recall models beat best un-normalized recall model by 12 points on hard sudoku and 2 points on hard chess.
• On prefix sums, normalized models are the only models achieving non-zero hard accuracy at the largest learning rate.
• Internal recall with outer normalization becomes competitive with external recall and substantially better on sudoku.

Surprising findings

• Internal recall without outer normalization has a much more anisotropic stable region than external recall, yet with outer normalization it matches or beats external recall on sudoku.
• Autonomous networks achieve 0% accuracy across all configurations on prefix sums.

Limitations

• Empirical analysis uses small single-layer transformers; scaling to larger models untested.
• Framework predicts necessary rather than sufficient conditions; relative ranking of internal vs external recall across tasks not resolved by theory.
• Theory relies on smoothness and transversality assumptions that are only argued to hold generically.

Experimental setup

Datasets: chess sudoku prefix-sums

Benchmarks: chess sudoku prefix-sums

Metrics: accuracy

Baselines: autonomous looped transformer external recall internal recall

Builds on: Deep Equilibrium Models Universal Transformer Adaptive Computation Time

Closest papers by meaning

• Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stopping (NeurIPS 2026 · similarity 0.91)
• On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding (ICML 2025 · similarity 0.89)
• Exact Expressive Power of Transformers with Padding (NeurIPS 2025 · similarity 0.87)
• Transformers Converge to Invariant Algorithmic Cores (NeurIPS 2026 · similarity 0.86)
• Sparse Layers are Critical to Scaling Looped Language Models (NeurIPS 2026 · similarity 0.86)
• Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning? (ICML 2024 · similarity 0.86)