Research map / Architectures and efficiency / Attention and transformers
LeaPformer: Enabling Linear Transformers for Autoregressive and Simultaneous Tasks via Learned Proportions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LeaPformer, which replaces cosFormer's sequence-length-dependent cosine re-weighting with a learned, token-level proportion module so linear transformers can be applied to autoregressive and simultaneous tasks.
Problem
Position-based re-weighting functions for linear transformers (e.g., cosFormer) require the target/total sequence length, which is unknown in autoregressive and simultaneous tasks, and their static patterns limit flexibility.
Approach
Generalize positional re-weighting to relative sequence proportions and learn these proportions per token with a compact two-layer feed-forward LeaP module (ReLU, sigmoid output) shared across heads, so no sequence length is needed.
Key results
- On Long-Range Arena, LeaPformer-1.5% reaches 58.69 average accuracy, second only to softmax attention (59.25), and beats all non-quadratic baselines.
- On WikiText-103 autoregressive LM, LeaPformer achieves 24.04 test perplexity, 0.13 better than cosFormer (24.17) but 2.37 worse than softmax (21.67).
- On MuST-C en-de SimulST, the decoder self-attention LeaPformer ablation gets 14.81 BLEU versus 15.07 for softmax (0.26 drop); on CoVoST fr-en the drop is 0.23 BLEU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add learned proportions to cosFormer | cosFormer · LRA average | accuracy: 57.43 -> 58.69 | Helped |
| Add learned proportions to cosFormer | cosFormer · LRA | RCP: 3.59 -> 5.20 | Helped |
| Add learned proportions to cosFormer | cosFormer · Wikitext-103b test | perplexity: 24.17 -> 24.04; 0.13 perplexity | Helped |
| Replace decoder self-attention with LeaPformer (ablation) | SimulST en-de · MuST-C en-de | BLEU: 15.07 -> 14.81; 0.26 BLEU | Hurt |
| Replace decoder self-attention with LeaPformer (ablation) | SimulST fr-en · CoVoST 2 fr-en | BLEU: 14.51 -> 14.28; 0.23 BLEU | Hurt |
| Increase cosFormer parameter count by 1.05x (ablation) | cosFormer · LRA tasks | accuracy | Mixed |
| Replace encoder self-attention with LeaPformer (ablation) | SimulST en-de · MuST-C en-de | BLEU: 15.07 -> 12.00 | Hurt |
| Replace cross-attention with LeaPformer (ablation) | SimulST en-de · MuST-C en-de | BLEU: 15.07 -> 13.95 | Hurt |
| Replace all attention blocks with LeaPformer (ablation) | SimulST en-de · MuST-C en-de | BLEU: 15.07 -> 11.19 | Hurt |
| Replace encoder self-attention with LeaPformer (ablation) | SimulST fr-en · CoVoST 2 fr-en | BLEU: 14.51 -> 11.18 | Hurt |
| Replace cross-attention with LeaPformer (ablation) | SimulST fr-en · CoVoST 2 fr-en | BLEU: 14.51 -> 13.25 | Hurt |
| Replace all attention blocks with LeaPformer (ablation) | SimulST fr-en · CoVoST 2 fr-en | BLEU: 14.51 -> 9.69 | Hurt |
Surprising findings
- Simply stepping the sequence length or using a max sequence length for cosFormer degraded en-de SimulST BLEU to around 1.1 in the authors' tests.
- S4D performed poorly on LRA (avg 49.09) in this implementation, contrary to published S4 results.
Limitations
- Gains on LRA are modest and variable across tasks (e.g., Pathfinder weaker than cosFormer for some settings).
- Still significantly below softmax attention on autoregressive language modeling (2.37 test perplexity gap).
- SimulST latency gains are small or negative (e.g., cross-attention slower than softmax).
Experimental setup
Datasets: Long-Range Arena WikiText-103 MuST-C en-de CoVoST 2 fr-en
Benchmarks: Long-Range Arena WikiText-103 MuST-C en-de tst-COMMON CoVoST 2 fr-en
Metrics: accuracy throughput perplexity BLEU RCP CA-AL
Baselines: Softmax Attention Linear Attention Linformer Performer BigBird Reformer Skyformer RoPE w/ Linear Attn. cosFormer ReLU Attention
Builds on: cosFormer Linear Transformer RoPE Skyformer framework wait-k policy
Closest papers by meaning
- Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks (ICML 2024 · similarity 0.89)
- How do Transformers Perform In-Context Autoregressive Learning ? (ICML 2024 · similarity 0.88)
- Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows (NeurIPS 2025 · similarity 0.87)
- Dynamic Convolutions Improve Transformers (NeurIPS 2026 · similarity 0.87)
- Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA (ICLR 2025 · similarity 0.87)
- Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.