Research map / Architectures and efficiency / Attention and transformers

Prompting a Pretrained Transformer Can Be a Universal Approximator

ICML 2024 · Architectures and efficiency › Attention and transformers · approach: Transformer expressivity and learnability · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows theoretically that prefix-tuning a fixed pretrained transformer can universally approximate sequence-to-sequence functions, with a single attention head sufficing for hypersphere functions and depth T+2 for general sequence functions, plus Jackson-type prefix-length bounds.

Problem

Whether prompting or prefix-tuning a fixed pretrained transformer can universally approximate continuous sequence-to-sequence functions, and how long the prefix must be to reach a given precision.

Approach

Shows that a single attention head with fixed H and W_V, restricted to hyperspheres, behaves like a vMF kernel expansion with prefixes as control points; a split-head to classical-head embedding (dimension 3(m+1)) plus a Kolmogorov-Arnold variant with Cantor-set inner function yields universality with depth T+2 and a Jackson-type prefix length bound.

Key results

• A single attention head of hidden dimension 3(m+1) with fixed H and W_V can approximate any continuous vector-valued function on S^m to arbitrary precision by prefixing.
• Sequence-to-sequence functions can be approximated by a transformer with T+2 attention layers, improving on prior meta-transformer constructions with exponential depth O(T eps^-m).
• Jackson-type prefix length bound N = O(eps^(-10-14m-4m^2)) for smooth targets, which the authors note suggests prefix-tuning may be less efficient than training.

Surprising findings

• Prefix-tuning a single attention head is sufficient for universal approximation on the hypersphere.
• The prefix-length bound suggests prefix-tuning may be less parameter-efficient than training the transformer.

Limitations

• Requires highly specific fixed H and W_V that are unlikely in real pretrained models.
• Jackson bound is likely not tight, with N growing very steeply in epsilon and dimension.
• Does not characterize which functions real pretrained transformers can approximate via prefixing.

Experimental setup

Baselines: Wang et al. (2023) meta-transformer prefix-tuning construction

Builds on: prefix-tuning Kolmogorov-Arnold representation theorem universal approximation of transformers

Closest papers by meaning

• Fundamental Limits of Prompt Tuning Transformers: Universality, Capacity and Efficiency (ICLR 2025 · similarity 0.90)
• Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling (NeurIPS 2024 · similarity 0.89)
• Approximation Rate of the Transformer Architecture for Sequence Modeling (NeurIPS 2024 · similarity 0.87)
• Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators? (ICLR 2024 · similarity 0.87)
• Universal In-Context Approximation By Prompting Fully Recurrent Models (NeurIPS 2024 · similarity 0.87)
• Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilities (NeurIPS 2025 · similarity 0.87)