Research map / Architectures and efficiency / Attention and transformers
EL-Attention: Memory Efficient Lossless Attention for Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace multi-head attention with EL-Attention | Transformer 270M · SQuAD 1.1, beam search, fp16 | speedup: 2.7x | Helped |
| Replace multi-head attention with EL-Attention | BART-large · XSum, beam search, fp16 | speedup: 4.7x | Helped |
| Replace multi-head attention with EL-Attention | BART-large · CNN/DailyMail, beam search, fp16 | speedup: 5.0x | Helped |
| Replace multi-head attention with EL-Attention | GPT-2-small · CNN/DailyMail, beam search, fp16 | speedup: 1.8x | Helped |
| Replace multi-head attention with EL-Attention | GPT-2-medium · CNN/DailyMail, beam search, fp16 | speedup: 2.2x | Helped |
| Replace multi-head attention with EL-Attention | Transformer 270M · SQuAD 1.1, diverse beam search, fp16 | speedup: 2.8x | Helped |
| Replace multi-head attention with EL-Attention | BART-large · XSum, diverse beam search, fp16 | speedup: 4.6x | Helped |
| Replace multi-head attention with EL-Attention | BART-large · CNN/DailyMail, diverse beam search, fp16 | speedup: 5.3x | Helped |
| Replace multi-head attention with EL-Attention | Transformer 270M · SQuAD 1.1, greedy search, fp16 | speedup: 1.6x | Helped |
| Replace multi-head attention with EL-Attention | BART-large · XSum, greedy search, fp16 | speedup: 2.5x | Helped |
| Replace multi-head attention with EL-Attention | BART-large · CNN/DailyMail, greedy search, fp16 | speedup: 3.1x | Helped |
| Replace multi-head attention with EL-Attention | BART-large · CNN/DailyMail test, fp32 | ROUGE-1: 44.21 -> 44.21 | No clear effect |
Closest papers by meaning
- A Training-Free Sub-quadratic Cost Transformer Model Serving Framework with Hierarchically Pruned Attention (ICLR 2025 · similarity 0.88)
- Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (ICLR 2024 · similarity 0.88)
- Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning (ICLR 2025 · similarity 0.88)
- Reducing Transformer Key-Value Cache Size with Cross-Layer Attention (NeurIPS 2024 · similarity 0.87)
- Simple linear attention language models balance the recall-throughput tradeoff (ICML 2024 · similarity 0.87)
- Star Attention: Efficient LLM Inference over Long Sequences (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.