Research map / Architectures and efficiency / Attention and transformers

Learning Theory of Transformers: Local-to-Global Approximation via Softmax Partition of Unity

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Transformer expressivity and learnability · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proves that a shallow (2-block), dense, wide Transformer with softmax attention and sinusoidal positional encodings can approximate alpha-Holder functions on [0,1]^d and compact Riemannian manifolds via a globally supported softmax partition of unity, with near-minimax generalization O(n^{-2α/(2α+d)} log n).

Problem

Understanding the approximation and generalization capability of Transformers, particularly the role of softmax attention, for Hölder regression on Euclidean domains and Riemannian manifolds, given that prior ReLU-based partition-of-unity constructions need deep networks or explicit meshes.

Approach

Replace mesh-based ReLU partitions of unity with a softmax partition of unity over a finite set of covering centers; the softmax cancels the ||x||^2 term so the weights become affine dot-product attention, letting a 2-block Transformer perform local expert approximation and global softmax aggregation.

Key results

• A dense 2-block Transformer (H^1=P+2 heads, H^2=1) achieves uniform eps-approximation of alpha-Hölder functions on [0,1]^d with O(eps^{-d/alpha}) total parameters.
• ERM over the Transformer hypothesis class attains expected squared L2 error O(n^{-2alpha/(2alpha+d)} log n) on Euclidean domains.
• The same rates hold on d-dimensional compact Riemannian manifolds embedded in R^{dbar}, with the rate depending only on intrinsic dimension d.

Limitations

• Results restricted to alpha in (0,1] Hölder functions
• Parameter magnitudes grow as eps^{-2d/alpha} log(1/eps), and sequence length P grows as eps^{-d/alpha}
• Theory only; no empirical experiments reported in the provided text

Experimental setup

Metrics: uniform approximation error expected L2 generalization error covering number

Baselines: Gurevych et al. Takakura and Suzuki Havrilla and Liao Shi et al. Jiao et al.

Builds on: ReLU partition of unity approximation Shepard's interpolation normalized RBF networks Takakura and Suzuki Transformer approximation Jiao et al. Transformer approximation

Closest papers by meaning

• Unraveling the Gradient Descent Dynamics of Transformers (NeurIPS 2024 · similarity 0.87)
• Predictive Geometry of Hidden Trajectories in Transformers (NeurIPS 2026 · similarity 0.87)
• Global Convergence in Training Large-Scale Transformers (NeurIPS 2024 · similarity 0.87)
• Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators? (ICLR 2024 · similarity 0.86)
• In-context Learning of Linear Dynamical Systems with Transformers: Approximation Bounds and Depth-separation (NeurIPS 2025 · similarity 0.86)
• Nonparametric In-Context Learning under Growing Geometric Complexity: Minimax Optimality and Local Geometry-Adaptivity of Transformers (NeurIPS 2026 · similarity 0.86)