Research map / Architectures and efficiency / Attention and transformers
Luna: Linear Unified Nested Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| make P contextual (output of previous layer) instead of non-contextual learnable parameter (ablation) | Luna-16 · WMT'14 EN-DE test | BLEU: 25.2 -> 27.3 | Helped |
| use mean-pooled last-layer P instead of [CLS] for classification (ablation) | Luna-16 · LRA Retrieval | accuracy: 79.38 -> 80.22 | Helped |
| use softplus instead of elu+1 activation in causal attention (ablation) | Luna-32 · WMT'14 EN-DE test | BLEU: 27.6 -> 27.8 | Helped |
| tie W_k with W_v (parameter sharing) (ablation) | Luna-32, softplus · WMT'14 EN-DE test | BLEU: 27.8 -> 27.5 | Hurt |
| increase projected length from 16 to 32 (ablation) | Luna softplus · WMT'14 EN-DE test | BLEU: 27.6 -> 27.8 | Helped |
| increase projected dimension from 16 to 256 (ablation) | Luna · LRA | average accuracy: 61.46 -> 61.95 | Helped |
| untie parameters (full sets of W_Q, W_K, W_V) (ablation) | Luna-128 pretrained on 160GB · QNLI dev | accuracy: 91.5 -> 92.2 | Helped |
| replace softmax attention with Luna | Transformer (RoBERTa-base) · LRA | average accuracy: 59.24 -> 61.46 | Helped |
| replace softmax attention with Luna-128 in pretraining on 160GB | RoBERTa-base · QNLI dev | accuracy: 92.8 -> 92.2 | Hurt |
| replace softmax attention with Luna-128 in pretraining on 16GB | RoBERTa-base · QNLI dev | accuracy: 90.9 -> 91.2 | Helped |
| replace softmax attention with Luna-16 in WMT'14 EN-DE | Transformer-base · WMT'14 EN-DE test | BLEU: 28.3 -> 27.6 | Hurt |
Closest papers by meaning
- ZeroS: Zero‑Sum Linear Attention for Efficient Transformers (NeurIPS 2025 · similarity 0.88)
- Twilight: Adaptive Attention Sparsity with Hierarchical Top-$p$ Pruning (NeurIPS 2025 · similarity 0.88)
- Gated Linear Attention Transformers with Hardware-Efficient Training (ICML 2024 · similarity 0.88)
- Theory, Analysis, and Best Practices for Sigmoid Self-Attention (ICLR 2025 · similarity 0.87)
- RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling (NeurIPS 2025 · similarity 0.87)
- MVA: Linear Attention with High-order Query-Keys Integration and Multi-level Vocabulary Decomposition (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.