Research map / Architectures and efficiency / Attention and transformers
Lite Transformer with Long-Short Range Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace standard Transformer blocks with LSRA Lite Transformer | Transformer · IWSLT'14 De-En | BLEU: +3.1 | Helped |
| Replace standard Transformer blocks with LSRA Lite Transformer | Transformer · WMT'14 En-De | BLEU: +1.2 | Helped |
| Replace standard Transformer blocks with LSRA Lite Transformer | Transformer · WMT'14 En-Fr | BLEU: +1.7 | Helped |
| Replace standard Transformer blocks with LSRA Lite Transformer | Transformer · WMT'14 En-De | BLEU: +0.5 | Helped |
| Replace standard Transformer blocks with LSRA Lite Transformer | Transformer · WMT'14 En-Fr | BLEU: +1.5 | Helped |
| Use LSRA architecture instead of Transformer | Transformer · CNN-DailyMail | Rouge-1 F1 | Hurt |
| Use LSRA architecture instead of Transformer | Transformer · CNN-DailyMail | Rouge-2 F1 | Hurt |
| Use LSRA architecture instead of Transformer | Transformer · CNN-DailyMail | Rouge-L F1 | No clear effect |
| Use Lite Transformer architecture with adaptive inputs | Adaptive Inputs · WIKITEXT-103 validation | perplexity: 3.9 lower | Helped |
| Use Lite Transformer architecture with adaptive inputs | Adaptive Inputs · WIKITEXT-103 test | perplexity: 1.8 lower | Helped |
| Use Lite Transformer instead of Evolved Transformer | Evolved Transformer · WMT'14 En-De mobile setting | BLEU: +0.5 | Helped |
| Pair Lite Transformer with pruning and quantization | Lite Transformer · WMT'14 En-Fr | model size compression: 18.2× | Mixed |
Closest papers by meaning
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.86)
- Trainable Transformer in Transformer (ICML 2024 · similarity 0.85)
- On the Power of Convolution-Augmented Transformer (AAAI 2025 · similarity 0.85)
- Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models (ICML 2024 · similarity 0.85)
- Core Context Aware Transformers for Long Context Language Modeling (ICML 2025 · similarity 0.85)
- Rodimus*: Breaking the Accuracy-Efficiency Trade-Off with Efficient Attentions (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.