Research map / Architectures and efficiency / Attention and transformers
Linear Attention for Efficient Bidirectional Sequence Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends Linear Transformers to bidirectional sequence modeling via LION, which gives full-attention, bidirectional RNN, and chunkwise forms, instantiated as LION-LIT, LION-RetNet and LION-S.
Problem
Linear Transformers are well studied for causal sequence modeling, but no unified framework exists for applying them to bidirectional sequence modeling, where bidirectional SSMs like Vim and Hydra rely on causal forms run in both directions and train slowly.
Approach
Define a bidirectional decay mask so that linear attention can be trained in full-attention parallel form while equivalent forward and backward RNN recurrences (with a half-diagonal correction to avoid double counting) support efficient inference, plus a chunkwise form; instantiated for no decay, fixed decay, and selective decay.
Key results
- On ImageNet-1K Small scale, LION-RotD reaches 79.9% top-1 vs DeiT 79.8% and Vim 80.3%, with about 10x faster training than Vim (relative time 1.66 vs 14.95).
- LION-S and LION-RetNet both solve Long Range Arena tasks, while LION-LIT without decay fails to.
- LION variants with 334M parameters closely match BERT and Hydra on MLM while being about 3x faster to train than Hydra.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multiple rotated token scans with LION-RetNet | LION-RetNet · ImageNet-1K Small | Top-1 accuracy (%): 73.5 -> 79.9 | Helped |
| Use multiple rotated token scans with LION-Selective | LION-Selective · ImageNet-1K Small | Top-1 accuracy (%): 74.0 -> 79.6 | Helped |
| Use multiple rotated token scans with LION-RetNet | LION-RetNet · ImageNet-1K Base | Top-1 accuracy (%): 77.8 -> 80.2 | Helped |
| Use multiple rotated token scans with LION-Selective | LION-Selective · ImageNet-1K Base | Top-1 accuracy (%): 76.3 -> 79.9 | Helped |
| Use LION-RetNet multi-scan instead of DeiT | ImageNet-1K Small · ImageNet-1K Small | Top-1 accuracy (%): 79.8 -> 79.9 | Helped |
| Use LION-RetNet multi-scan instead of DeiT | ImageNet-1K Small · ImageNet-1K Small | Train time (relative; lower is better): $\times$1 -> $\times$1.66 | Hurt |
| Use LION-RetNet multi-scan instead of DeiT | ImageNet-1K Base · ImageNet-1K Base | Top-1 accuracy (%): 81.8 -> 80.2 | Hurt |
| Use full attention rather than SSM training | LION · ImageNet-1K | Training speed: 10 \times | Helped |
| Use diagonal decay instead of no decay (ablation) | LRA · LRA | Task solvability | Helped |
| Remove decay (ablation) | LION-LinAtt · LRA | Task solvability | Hurt |
| Use larger chunk sizes (ablation) | LION-RetNet chunkwise inference · ImageNet-1K at 1248-resolution | Inference time | Helped |
| Use larger chunk sizes (ablation) | LION-RetNet chunkwise inference · ImageNet-1K at 1248-resolution | GPU memory: OOM beyond size 1024 | Hurt |
Surprising findings
- LION-LIT, the variant without decay, fails to solve Long Range Arena, showing decay is needed for long sequences.
Limitations
- LION-LIT without decay underperforms and cannot solve Long Range Arena tasks.
- Image-classification accuracy of LION-LIT and LION-RetNet is below DeiT at Base scale.
Experimental setup
Datasets: ImageNet-1K C4 GLUE Long Range Arena OpenWebText
Benchmarks: ImageNet-1K GLUE Long Range Arena OpenWebText
Metrics: top-1 accuracy relative training time memory perplexity
Baselines: ViT DeiT Vim Hydra BERT
Builds on: Linear Transformer RetNet Gated RFA Mamba-2 Vision Mamba
Closest papers by meaning
- Towards Understanding the Universality of Transformers for Next-Token Prediction (ICLR 2025 · similarity 0.88)
- How Transformers Learn Causal Structure with Gradient Descent (ICML 2024 · similarity 0.87)
- Selective Induction Heads: How Transformers Select Causal Structures in Context (ICLR 2025 · similarity 0.87)
- Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues (ICML 2024 · similarity 0.87)
- Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulation (ICLR 2025 · similarity 0.86)
- Flash Inference: Near Linear Time Inference for Long Convolution Sequence Models and Beyond (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.