Research map / Architectures and efficiency / Attention and transformers

Phase Space Attention: A Hairer Lift Resolves the Single-Layer Induction Obstruction

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Transformer expressivity and learnability

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Phase Space Attention, a symplectic upper-shear augmentation of post-RoPE query and key streams (Hairer lift of Störmer-Verlet), argued to escape the Sanford-Hsu-Telgarsky single-layer induction obstruction, with a closed-form two-branch induction phase transition and small-scale training experiments (4–92M, nano 54k params).

Problem

Single-layer attention cannot represent in-context induction under finite bit budgets (SHT lower bound), so induction normally needs two stacked layers; the paper asks whether a one-layer, one-step, symplectic score augmentation can make single-layer induction representable, particularly for sub-100

Approach

Lift attention to phase space by shifting post-RoPE query and key streams with the symplectic upper shear (q,p)->(q+γp,p), with p the first backward difference, giving score (q_t+γp_q,t)^T(k_j+γp_k,j); the shear raises joint filter order from (0,0) to (1,1), and within a factorised subclass it is the unique swap-symmetric, DC-neutral, high-pass causal choice.

Key results

• Key-only half-lift reaches 0.949 single-layer induction accuracy vs 0.811 for the symmetric operator at nano scale (max over coupling grid, three seeds).
• At 91.3M parameters with 12 layers, single-layer induction forms in 3/3 seeds at gamma=0.80 in a mean of 717 steps vs 2/3 seeds and 2700 steps at gamma=0 (reported as 3.8x faster).
• Nano-scale single-layer induction of 69.1±8.2% at gamma=2.0 over five seeds; Pearson r=0.9966 between the closed-form transfer function and measured response at gamma=2.

Surprising findings

• The key-side half-lift is more accurate than the symmetric shear on the induction task, so symmetry buys structure rather than accuracy.
• On randomised-offset induction the in-configuration optimum is gamma=0 at every depth tested, with the uncoupled baseline reaching 0.99 in 500 steps.

Limitations

• The closed-form two-branch phase transition is derived in a T4-only Gaussian reduction with a fitted calibration constant; restoring the T3 channel shifts gamma_c substantially, so the law is not robust.
• Empirical validation is limited to small models (nano 54k params, up to ~92M) on synthetic associative recall; no language-model-scale gain demonstrated.
• Transfer from reduction to trained models is approximate; the paper itself flags the branch structure as reduction-sensitive.

Experimental setup

Datasets: synthetic associative recall synthetic induction sequences

Benchmarks: synthetic associative recall single-layer induction task

Metrics: test accuracy Pearson correlation with pre-softmax Bode transfer function optimizer steps to emergence chance-normalised accuracy

Baselines: standard causal attention query-only shear key-only shear asymmetric shear Differential Transformer (first-order)

Builds on: Rotary position embedding (RoPE) Stormer-Verlet integrator Hairer geometric numerical integration Sanford-Hsu-Telgarsky induction bound Differential Transformer

Closest papers by meaning

• On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.86)
• Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobians (NeurIPS 2025 · similarity 0.85)
• Spectral Conditioning of Attention Improves Transformer Performance (NeurIPS 2025 · similarity 0.85)
• Block-Wise Differentiable Sinkhorn Attention: Tail-Refinement Gradients with a Gap-Aware Dustbin Bridge (NeurIPS 2026 · similarity 0.84)
• The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training (ICML 2025 · similarity 0.84)
• Optimality and NP-Hardness of Transformers in Learning Markovian Dynamical Functions (NeurIPS 2025 · similarity 0.84)