Research map / Architectures and efficiency / Attention and transformers
Vocabulary In-Context Learning in Transformers: Benefits of Positional Encoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that single-layer Transformers doing in-context learning over a finite vocabulary cannot reach the universal approximation property without positional encoding, but can with suitable (dense, unbounded) positional encodings.
Problem
Whether in-context learning in single-layer Transformers, where context tokens come from a finite vocabulary, can achieve the universal approximation property (UAP), and what role positional encoding plays.
Approach
Treats context as a control parameter so a single-layer attention block reduces to a one-hidden-layer FNN; without positional encoding, finite-vocabulary contexts give a finite-dimensional function span (non-UAP), while adding positional encodings whose shifted vocabulary is dense lets constructed contexts approximate any continuous function, with weaker density requirements for ReLU and exponenti
Key results
- Single-layer Transformers with finite vocabulary and no positional encoding (T*) cannot achieve the UAP, shown by a counting-zeros argument via Proposition on exponential sums.
- With positional encoding whose shifted vocabulary set is dense in R^{d_x} and P_y=0, single-layer Transformers with element-wise activations achieve the UAP (Theorem on T_{*,P}).
- Informal result: density of S only in [-1,1]^{d_x} suffices for ReLU, and density only in a neighborhood suffices for exponential activation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| restrict context tokens to a finite vocabulary without positional encoding | single-layer Transformer VICL · VICL function approximation | universal approximation property (UAP) | Hurt |
| add absolute positional encoding satisfying the theorem's density and vocabulary conditions | single-layer Transformer VICL · VICL function approximation | universal approximation property (UAP) | Helped |
| restrict the positional-encoding density condition to S dense in [-1, 1]^{d_x} | single-layer ReLU Transformer VICL · VICL function approximation | universal approximation property (UAP) | Helped |
| use exponential activation with S dense only in a neighborhood of a point | single-layer exponential Transformer VICL · VICL function approximation | universal approximation property (UAP) | Helped |
| allow context tokens to come from the entire vector space rather than a finite vocabulary | single-layer Transformer ICL · ICL function approximation | universal approximation property (UAP) | Helped |
Limitations
- Analysis limited to single-layer Transformers with absolute positional encodings and element-wise activations; softmax extension is partial.
- Requires unbounded, dense positional encodings and a sparse partition assumption on Q, K, V (F=0, block structure) that may not hold in practice.
- Results are existence/approximation-theoretic; no experiments validate the constructions.
Experimental setup
Builds on: universal approximation theorem for FNNs Kronecker approximation theorem Stone-Weierstrass theorem
Closest papers by meaning
- How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations (ICLR 2024 · similarity 0.90)
- Transformers are Universal In-context Learners (ICLR 2025 · similarity 0.88)
- Understanding the Generalization of In-Context Learning in Transformers: An Empirical Study (ICLR 2025 · similarity 0.88)
- Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context (NeurIPS 2025 · similarity 0.88)
- Functional Interpolation for Relative Positions improves Long Context Transformers (ICLR 2024 · similarity 0.88)
- Transformers are Minimax Optimal Nonparametric In-Context Learners (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.