Research map / Architectures and efficiency / Attention and transformers
Pause Tokens Strictly Increase the Expressivity of Constant-Depth Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically that pause tokens strictly increase the expressivity of constant-depth, log-width Transformers (AC0 with constant precision, TC0 with log precision), and empirically that 2-layer causal Transformers learn parity only with pause tokens.
Problem
Whether pause tokens (filler tokens such as "...") merely aid optimisation or fundamentally increase Transformer computational expressivity, which remains unexplained theoretically.
Approach
Characterises constant-depth, logarithmic-width Transformers with polynomially many or quasi-polynomially many pause tokens via circuit classes: constant-precision with pause tokens equals AC0 (strictly more than without), and log-precision with pause tokens equals TC0; the construction simulates circuit layers with pairs of attention layers, using pause tokens as gate/argument workspace.
Key results
- Constant-precision Transformers with polynomially many pause tokens equal AC0 (TF[1,L,P] = AC0), while without pause tokens they are a strict subset of AC0 in the non-uniform setting.
- Logarithmic-precision Transformers with pause tokens are shown equivalent to TC0 (TF[L,L,P] = TC0).
- In the parity experiment, a 2-layer causal Transformer with pause tokens learns parity well beyond the causal-only baseline, which falls to chance around 100-bit sequences; the non-causal model stays near-perfect until about 200 bits.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add n pause tokens with causal masking | 2-layer GPT-2-style Transformer · Parity on bit sequences | test accuracy | Helped |
| remove causal masking | 2-layer GPT-2-style Transformer without pause tokens · Parity on bit sequences | test accuracy | Helped |
| use causal masking without pause tokens | 2-layer GPT-2-style Transformer · Parity on bit sequences | test accuracy | Hurt |
| provide only final-prediction loss, without hint labels | shallow Transformer in all regimes · Parity | test accuracy | Hurt |
| add parallel threshold-value hint labels | non-causal and pause-token Transformers · Parity | learnability | Helped |
| add pause tokens to a constant-precision, logarithmic-width Transformer | Constant-precision Transformer · Boolean function computation | expressivity | Helped |
| add pause tokens to a logarithmic-precision Transformer | Logarithmic-precision Transformer · Boolean function computation | expressivity | Helped |
Surprising findings
- Without pause tokens, causal Transformers trained on parity fall to chance by length 100, and the authors could not prove a strict separation in the log-precision (TC0) regime.
Limitations
- Strict separation for log-precision (TF[L,L,0] vs TF[L,L,P]) is not proven due to lack of superpolynomial TC0 lower bounds.
- Uniform fixed-depth separation requires quasi-polynomial pause tokens.
- Separation depends on saturation arithmetic; modular arithmetic would move the class to ACC0.
Experimental setup
Datasets: synthetic random bit strings for parity
Benchmarks: synthetic parity accuracy across sequence lengths
Metrics: test accuracy
Baselines: Transformers without pause tokens non-causal Transformer
Builds on: pause tokens (Goyal et al.) chain-of-thought circuit complexity of Transformers (Hahn; Merrill and Sabharwal)
Closest papers by meaning
- A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers (NeurIPS 2025 · similarity 0.89)
- Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding (NeurIPS 2025 · similarity 0.89)
- Distilling Sequential Computation in Transformer Language Models (NeurIPS 2026 · similarity 0.89)
- Characterizing the Expressivity of Fixed-Precision Transformer Language Models (NeurIPS 2025 · similarity 0.88)
- Exact Expressive Power of Transformers with Padding (NeurIPS 2025 · similarity 0.88)
- The Expressive Power of Transformers with Chain of Thought (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.