Research map / Architectures and efficiency / Attention and transformers
On the Power of Convolution-Augmented Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Convolution-Augmented Transformer (CAT), which adds short convolutions to the K/Q/V embeddings of attention, proving it solves N-gram associative recall and selective copying with one layer and length generalization, plus a landmark-convolution sparse attention variant.
Problem
Transformers lack locality without positional encoding and struggle with length generalization, while SSMs/Mamba lack recall; the paper asks what convolution-attention synergy provably buys on recall, copying, and length generalization tasks.
Approach
Insert learnable 1D convolutional filters into the key, query and value embeddings of a single attention layer; theory shows the local convolution plus global attention solves N-gram AR and selective copy with one layer and generalizes in length, and convolutional summarization (landmark tokens) enables sparse attention.
Key results
- Single-layer CAT maintains 100% accuracy on MQAR across model dimensions and sequence lengths, while Mamba and Based degrade at longer lengths or smaller dimensions.
- A 370M CAT with RoPE (MH-CAT RoPE) reaches average zero-shot accuracy 0.422 vs Pythia 0.405 and Wikitext perplexity 25.858 vs 27.410.
- CAT without positional encoding stays competitive (avg acc 0.393) while Pythia without PE diverges during training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 1D convolution to Pythia without positional embedding | CAT · Wikitext | perplexity: 27.410 -> 29.216 | Hurt |
| add 1D convolution and RoPE to Pythia | CAT · Wikitext | perplexity: 27.410 -> 26.776 | Helped |
| use multi-head convolution and RoPE | CAT · Average downstream accuracy | accuracy: 0.405 -> 0.422 | Helped |
| add RoPE to MH-CAT (ablation) | MH-CAT · Wikitext | perplexity: 27.417 -> 25.858 | Helped |
| add RoPE to MH-CAT (ablation) | MH-CAT · Lambada_std | perplexity: 58.959 -> 47.593 | Helped |
| add RoPE to CAT (ablation) | CAT · Lambada_openai | perplexity: 42.260 -> 38.557 | Mixed |
| use multi-head convolution instead of 1D CAT convolution, both with RoPE (ablation) | CAT · Average downstream accuracy | accuracy: 0.407 -> 0.422 | Helped |
| use multi-head convolution instead of 1D CAT convolution, both without PE (ablation) | CAT · Wikitext | perplexity: 29.216 -> 27.417 | Helped |
| remove positional encoding from Pythia (ablation) | Pythia · language modeling training | convergence | Hurt |
| remove K/Q convolution from CAT (ablation) | CAT · real language modeling | performance | No clear effect |
| use short convolution in the single-layer CAT model | CAT · MQAR and MQNAR | accuracy | Helped |
| train on sequence length 128 and test across lengths 32 to 1,024 | CAT · MQAR length generalization | accuracy | Helped |
Surprising findings
- Pythia without positional encoding diverges to extremely large losses, while CAT without PE still trains competitively.
- K/Q convolution does not give noticeable real-data gains despite helping the theoretical N-gram AR construction, which the authors attribute to possible dilution of attention scores.
Limitations
- Experiments focus only on pretraining; finetuning is left as future work.
- K/Q convolution gives no noticeable real-data benefit despite theoretical role.
- Landmark CAT sparse attention is not evaluated on real language modeling tasks.
Experimental setup
Datasets: SlimPajama synthetic MQAR/MQNAR datasets
Benchmarks: Wikitext Lambada PIQA HellaSwag Winogrande ARC-Easy ARC-Challenge MQAR MQNAR selective copying
Metrics: accuracy perplexity
Baselines: Mamba Based linear attention attention-only Transformer Pythia TF++ GLA
Builds on: Pythia Multi-query associative recall (Zoology) Mamba Landmark Attention ALiBi
Closest papers by meaning
- Dynamic Convolutions Improve Transformers (NeurIPS 2026 · similarity 0.90)
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.89)
- Curse of High Dimensionality Issue in Transformer for Long Context Modeling (ICML 2025 · similarity 0.88)
- Forgetting Transformer: Softmax Attention with a Forget Gate (ICLR 2025 · similarity 0.88)
- A Training-Free Sub-quadratic Cost Transformer Model Serving Framework with Hierarchically Pruned Attention (ICLR 2025 · similarity 0.88)
- SEA: Sparse Linear Attention with Estimated Attention Mask (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.