Research map / Architectures and efficiency / Attention and transformers
Unlimiformer: Long-Range Transformers with Unlimited Length Input
Amanda Bertsch, Uri Alon, Graham Neubig, Matthew R. Gormley
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Unlimiformer at test time without additional training | BART-base · GovReport | ROUGE-1: 48.7 -> 49.7 | Improved |
| Use Unlimiformer for validation and early stopping (ablation) | BART-base · GovReport | ROUGE-1: 48.7 -> 51.0 | Improved |
| Train on non-overlapping chunks, then add test-time Unlimiformer (ablation) | BART-base · GovReport | ROUGE-1: 46.2 -> 53.4 | Improved |
| Train on non-overlapping chunks without test-time Unlimiformer (ablation) | BART-base · GovReport | ROUGE-1: 48.7 -> 46.2 | Hurt |
| Apply test-time SLED instead of standard finetuning | BART-base · GovReport | ROUGE-1: 48.7 -> 45.8 | Hurt |
| Add Unlimiformer with alternating training | BART-base · GovReport | ROUGE-1: 55.2 -> 56.6 | Improved |
| Add Unlimiformer with alternating training | PRIMERA · SummScreen | ROUGE-1: 33.0 -> 33.3 | Improved |
| Apply Unlimiformer with retrieval training | PRIMERA · BookSum | EntMent: 11.6 -> 25.5 | Improved |
| Apply Unlimiformer only at test time | BART-base · BookSum | EntMent: 10.0 -> 21.9 | Improved |
| Use random-encoded rather than retrieval training (ablation) | BART + Unlimiformer · BookSum | ROUGE-1: 36.8 -> 37.3 | Mixed |
| Use random-encoded rather than retrieval training (ablation) | BART + Unlimiformer · BookSum | ROUGE-2: 8.3 -> 6.7 | Hurt |
| Increase the length of input available to Unlimiformer (ablation) | Unlimiformer · BookSum | EntMent | Improved |
Closest papers by meaning
- IceFormer: Accelerated Inference with Long-Sequence Transformers on CPUs (ICLR 2024 · similarity 0.86)
- StableMask: Refining Causal Masking in Decoder-only Transformer (ICML 2024 · similarity 0.86)
- A Training-Free Sub-quadratic Cost Transformer Model Serving Framework with Hierarchically Pruned Attention (ICLR 2025 · similarity 0.86)
- InfoFlow: A Framework for Multi-Layer Transformer Analysis (NeurIPS 2026 · similarity 0.86)
- Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.85)
- Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
