Research map / Architectures and efficiency / Attention and transformers

Nonparametric In-Context Learning under Growing Geometric Complexity: Minimax Optimality and Local Geometry-Adaptivity of Transformers

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Training dynamics of in-context learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies nonparametric in-context learning over growing heterogeneous manifold mixtures, proving an aggregate minimax rate, an oracle tangent local-polynomial estimator attaining it, and a softmax transformer that realizes it with logarithmic depth and polynomial size.

Problem

Nonparametric in-context regression when the data lie on an unknown, sample-size-dependent mixture of manifolds with heterogeneous dimensions, smoothness, and masses; prior ICL theory covered Euclidean domains or a single manifold.

Approach

Combine a polynomial-graph tangent estimate with local-polynomial regression in estimated tangent coordinates, and show a two-stage softmax/ReLU transformer with chartwise reduced solvers can implement this estimator in one forward pass, yielding an aggregate minimax rate sum_k pi_k (n pi_k)^{-2a_k/(2a_k+d_k)}.

Key results

• Transformer test MSE falls from 14.455e-3 at 100k pretraining prompts to 0.505e-3 at 1M prompts, versus 0.690e-3 for LP-CV.
• At 1M prompts the transformer's MSE is 26.75% lower than LP-CV, with paired risk ratio 0.733 (CI [0.686, 0.782]).
• Theory: minimax lower bound and matching oracle upper bound of order R_n; transformer approximation error n^{-A_0} with logarithmic depth and polynomial size.

Limitations

• Empirical transformer is a learned softmax workspace model, not the residual-ReLU construction proved in theory.
• Theoretical resource bounds are conservative; finite-sample constant conditions not certified at n=40,000.
• Experiments use a single training seed and LP-CV uses the true tangent, so the comparison is not against an estimated-tangent baseline.

Experimental setup

Benchmarks: synthetic heterogeneous mixture-of-manifolds ICL task

Metrics: mean squared error relative risk ratio vs LP-CV

Baselines: LP-CV (geometry-informed local-polynomial with true tangent)

Builds on: local polynomial regression nonparametric in-context learning theory

Closest papers by meaning

• Transformers are Minimax Optimal Nonparametric In-Context Learners (NeurIPS 2024 · similarity 0.90)
• In-Context Learning with Representations: Contextual Generalization of Trained Transformers (NeurIPS 2024 · similarity 0.89)
• Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions (ICLR 2024 · similarity 0.88)
• In-context Convergence of Transformers (ICML 2024 · similarity 0.88)
• Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context (NeurIPS 2025 · similarity 0.87)
• In-Context Learning through the Bayesian Prism (ICLR 2024 · similarity 0.87)