Research map / Architectures and efficiency / Attention and transformers

Scaling Limits of Long-Context Transformers

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse and structured attention · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Models softmax attention with a fixed query and n i.i.d. random keys on the sphere, and derives the critical inverse-temperature scale (n^{2/(d-1)}) and limit laws for ordered attention weights and outputs across sub-, critical and supercritical regimes.

Problem

Long-context softmax attention either averages uniformly or collapses to a single key as the inverse temperature grows with context length n; the question is which scalings of beta_n yield non-degenerate limiting laws for attention weights and outputs.

Approach

Treats the distance-to-query (D2Q) of random keys as the random variable; the critical scale is set by the local exponent of the D2Q CDF near zero (F(t) ~ t^{1/alpha}, alpha=2/(d-1)), yielding a Poisson point process limit for the ordered weights at beta_n ~ n^alpha and a backward-heat/density-gradient interpretation of subcritical outputs.

Key results

• Critical inverse temperature scales as beta_n ~ n^{2/(d-1)} for uniform keys on S^{d-1}; the subcritical/critical/supercritical regimes are characterized by theorems on ordered weights.
• At criticality, ordered weights converge to normalized atoms of a Poisson point process with intensity C(q)(y/gamma)^{1/alpha}.
• Numerical experiments on uniform contexts on S^4 (n=1000) show the rescaled ratios A_(k)/A_(1) follow the predicted exp(-x^alpha) curve in the subcritical regime.

Surprising findings

• With beta_n=1 (no rescaling), a key with a constant alignment margin still receives weight Theta(1/n), so attention cannot select relevant keys without temperature growth.

Limitations

• Analysis assumes i.i.d. keys (with extension to weak dependence and RoPE claimed only in appendices) and a fixed query with a C^2 density.
• Numerical validation is limited to simulations with n up to 10^4 and a few specific densities.
• Scaling results are asymptotic and do not directly address trained transformer models.

Experimental setup

Metrics: mean of A_(1) over trials ratio A_(k)/A_(1) vs prediction exp(-x^alpha) output displacement field visualization

Builds on: Random Energy Model Jordan-Kinderlehrer-Otto gradient flow PostLN self-attention RoPE

Closest papers by meaning

• Selective Attention: Enhancing Transformer through Principled Context Control (NeurIPS 2024 · similarity 0.88)
• Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers (ICML 2025 · similarity 0.88)
• Infinite-Width Limit of a Single Attention Layer: Analysis via Tensor Programs (NeurIPS 2025 · similarity 0.88)
• High-Dimensional Learning Dynamics of Attention-Indexed Models (NeurIPS 2026 · similarity 0.87)
• How Smooth Is Attention? (ICML 2024 · similarity 0.87)
• Self-attention Networks Localize When QK-eigenspectrum Concentrates (ICML 2024 · similarity 0.87)