Research map / Architectures and efficiency / Attention and transformers
Two Heads are Better than One: Simulating Large Transformers with Small Ones
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that a large transformer with input length N can be simulated by O((N/M)^2) small transformers of input length M (tight in the worst case), and by only O(N/M) under average-case, sliding-window, or attention-sink assumptions.
Problem
Long input sequences make self-attention quadratic, while hardware is optimized for short sequences; the question is whether large transformers can be simulated using multiple small transformers and how many calls are needed.
Approach
Treats small transformers as oracles and decomposes the N x N attention into M x M blocks, using synthetic tokens (constant-valued normalizers) to recover block-wise softmax numerators and denominators; under bounded-score or random-permutation assumptions, only O(N/M) oracle calls are needed, and sliding-window/attention-sink structure also yields linear calls.
Key results
- Any N-length transformer can be simulated with O((N/M)^2 * HL/(H'L')) small-transformer oracle calls, and this quadratic count is shown to be necessary in the worst case under standard complexity assumptions.
- Under bounded query/key/value assumptions (average-case), O(N/M * HL/(H'L')) oracle calls suffice to obtain a (1+eps) approximation with probability at least 0.9.
- Sliding window and attention-sink transformers can be simulated with O(N/M) oracle calls even in the worst case.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use small transformers to simulate a large transformer on arbitrary inputs | Large transformer with L layers and H heads · Arbitrary inputs, standard attention | oracle calls: O((\frac{N}{M})^2\cdot\frac{HL}{H'L'}) | Helped |
| Restrict both transformers to causal masking | Small-transformer simulation · Arbitrary inputs, causal masking | oracle calls: O((\frac{N}{M})^2\cdot\frac{HL}{H'L'}) | No clear effect |
| Assume bounded attention scores and bounded contribution magnitudes | Small-transformer simulation · Inputs satisfying the stated boundedness conditions | oracle calls: O((\frac{N}{M})^2\cdot\frac{HL}{H'L'}) -> O(\frac{N}{M}\cdot \frac{HL}{H'L'}) | Helped |
| Use constant-size sliding-window masking | Small-transformer simulation · Constant-size sliding window | oracle calls: O((\frac{N}{M})^2\cdot\frac{HL}{H'L'}) -> O(\frac{N}{M}\cdot\frac{HL}{H'L'}) | Helped |
| Add constant-size attention sinks to sliding-window attention | Sliding-window small-transformer simulation · Constant-size sliding window with constant-size attention sinks | oracle calls: O(\frac{N}{M}\cdot\frac{HL}{H'L'}) | No clear effect |
| Use N/M small transformers to simulate a large transformer in the worst case | Single-head, single-layer transformers · Worst-case inputs | expressive power | Hurt |
Surprising findings
- Linear numbers of small transformers are strictly weaker than one large transformer for worst-case inputs (converse result for single-head transformers), requiring extra small matrix multiplications to match.
Limitations
- Worst-case simulation requires quadratic oracle calls, so no savings in total floating-point operations over the oracles.
- Linear-call results rely on bounded-score or average-case assumptions that may not hold in practice.
- Purely theoretical; no experiments are reported.
Experimental setup
Baselines: Hierarchical Transformers Reformer Longformer Performer Mamba
Builds on: StreamingLLM Hierarchical Transformers
Closest papers by meaning
- Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling (NeurIPS 2024 · similarity 0.88)
- IceFormer: Accelerated Inference with Long-Sequence Transformers on CPUs (ICLR 2024 · similarity 0.88)
- A Formal Framework for Understanding Length Generalization in Transformers (ICLR 2025 · similarity 0.88)
- Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators? (ICLR 2024 · similarity 0.87)
- When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspective (NeurIPS 2025 · similarity 0.87)
- Arithmetic Transformers Can Length-Generalize in Both Operand Length and Count (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.