Research map / Architectures and efficiency / Attention and transformers

Transformers are Universal In-context Learners

ICLR 2025 · Architectures and efficiency › Attention and transformers · approach: Transformer expressivity and learnability · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that deep transformers with fixed embedding dimension and head count proportional to dimension are universal approximators of continuous in-context mappings over probability measures of tokens, for arbitrarily many tokens, in both unmasked and causal settings.

Problem

Expressivity of transformers as in-context mappings over an arbitrarily large (even infinite) number of context tokens, which prior universality results only handled with embedding dimension growing with token count.

Approach

Represents token sets as probability measures (empirical measures for finite n), defines continuity via weak* / Wasserstein topology, and proves universality using a Stone-Weierstrass argument on cylindrical functions built from single-head 1-D attention; masked (causal) case handled via space-time lifting with Wasserstein-Lipschitz contexts.

Key results

• Theorem 1 (unmasked): for any compact token domain and continuous in-context map over weak*-topology measures, a deep transformer with per-layer embedding dimension at most d+3d' and at most d' heads of size 1 approximates it to any precision, independent of n.
• Theorem 2 (masked): the same universality holds for causal identifiable in-context maps restricted to Wasserstein-Lipschitz space-time contexts.

Limitations

• Results are non-quantitative: no explicit control of MLP parameter dependence on precision epsilon.
• Number of heads grows proportionally to output dimension, with each head of dimension 1.
• Masked setting requires identifiability and Wasserstein-Lipschitz context assumptions.

Experimental setup

Builds on: Transformers as measure-to-measure mappings Stone-Weierstrass theorem

Closest papers by meaning

• In-context Convergence of Transformers (ICML 2024 · similarity 0.90)
• Transformers are Minimax Optimal Nonparametric In-Context Learners (NeurIPS 2024 · similarity 0.89)
• Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers (NeurIPS 2024 · similarity 0.88)
• Vocabulary In-Context Learning in Transformers: Benefits of Positional Encoding (NeurIPS 2025 · similarity 0.88)
• Towards Understanding How Transformers Learn In-context Through a Representation Learning Lens (NeurIPS 2024 · similarity 0.88)
• Towards Understanding the Universality of Transformers for Next-Token Prediction (ICLR 2025 · similarity 0.88)