Research map / Architectures and efficiency / Attention and transformers
MVA: Linear Attention with High-order Query-Keys Integration and Multi-level Vocabulary Decomposition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Converts pre-trained Softmax-attention LLMs (Mistral-7B) into linear-attention models via a unified high-order QK integration (MetaLA + GSA) and multi-level vocabulary decomposition, optionally with a soft-integrated sliding window (MVA-SW), using LoRA fine-tuning on SlimPajama and Alpaca/RedPajama.
Problem
Training linear attention LLMs from scratch is expensive and lags Softmax attention; converting pre-trained Softmax LLMs to linear attention with minimal tokens still leaves performance gaps and fixed-size states lose information.
Approach
Combines a low-order MetaLA-style branch (low-frequency) with a high-order GSA-style branch (high-frequency) via High-order QK Integration, adds multi-level vocabulary decomposition that recursively compresses the compression error into additional states to expand memory, and softly fuses a sliding window by attention-score-based weighting.
Key results
- MVA-SW fine-tuned from Mistral-7B with 0.1B tokens reaches 66.1 average vs 66.6 for original Mistral-7B and improves MMLU to 57.1 over GSA-SW's 54.5.
- MVA without sliding window, fine-tuned with 10B tokens, reaches 60.3 average vs GSA's 57.7 and surpasses GSA on all listed tasks.
- On MMLU with 0.02B tokens, MVA-SW scores 52.6 vs LoLCATs 51.4 at 0.04B tokens.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine MVA with sliding-window attention using soft integration (ablation) | Mistral-7B · MMLU | accuracy: 53.2 -> 57.1 | Helped |
| remove sliding-window attention (ablation) | MVA-SW · MMLU | accuracy: 57.1 -> 25.0 | Hurt |
| remove the MVA component and use sliding-window attention only (ablation) | MVA-SW · MMLU | accuracy: 57.1 -> 51.9 | Hurt |
| replace hard integration with soft integration (ablation) | MVA-SW · MMLU | accuracy: 53.2 -> 57.1 | Helped |
| integrate the GSA and MetaLA branches instead of using only the MetaLA order (ablation) | MVA · ARC-Challenge | accuracy: 0.3527 -> 0.3857 | Helped |
| add multi-level vocabulary decomposition (ablation) | MVA · ARC-Challenge | accuracy: 0.3857 -> 0.4044 | Helped |
| add multi-level vocabulary decomposition (ablation) | MVA · ARC-Easy | accuracy: 0.7075 -> 0.7391 | Helped |
| replace the GSA Softmax with a sigmoid perceptron (ablation) | MVA · ARC-Challenge | accuracy: 0.3389 -> 0.3407 | No clear effect |
| increase MVA fine-tuning from +2.5B to +10B tokens | MVA · Mistral-7B evaluation tasks | Avg.: 57.2 -> 60.3 | Helped |
| increase MVA fine-tuning from +2.5B to +10B tokens | MVA · MMLU | accuracy: 30.2 -> 34.4 | Helped |
| increase MVA-SW fine-tuning from +0.1B to +2B tokens | MVA-SW · MMLU | accuracy: 57.1 -> 57.3 | No clear effect |
| convert Mistral-7B to MVA-SW instead of using LoLCATs | Mistral-7B · MMLU | accuracy: 51.4 -> 52.6 | Helped |
Surprising findings
- MVA is slightly slower in generation latency than GSA despite comparable memory, due to vocabulary decomposition overhead.
- Freezing QKV weights and training only the linear part for 2B tokens slightly improved performance, per the authors.
Limitations
- Training gradient memory and time per iteration higher with vocabulary decomposition (105.79 s/it vs 82.18 s/it for MVA w/o VD).
- Per-head parameters not introduced; shared parameters across heads are a stated limitation for future work.
- Sliding window requires training length of 1.5K due to GPU memory constraints.
Experimental setup
Datasets: SlimPajama AlpacaClean RedPajama
Benchmarks: ARC-e ARC-c HellaSwag PIQA Winogrande NQ TriviaQA MMLU
Metrics: accuracy average accuracy inference latency memory usage
Baselines: GSA GLA RetNet SUPRA LoLCATs RWKV6 Mamba Llama2 Gemma Mistral-7B
Builds on: Gated Slot Attention MetaLA Gated Linear Attention Delta rule LoLCATs
Closest papers by meaning
- MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map (NeurIPS 2024 · similarity 0.90)
- Efficient Low Rank Attention for Long-Context Inference in Large Language Models (NeurIPS 2025 · similarity 0.89)
- More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding (NeurIPS 2026 · similarity 0.89)
- Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free (NeurIPS 2025 · similarity 0.89)
- Simple linear attention language models balance the recall-throughput tradeoff (ICML 2024 · similarity 0.89)
- Loki: Low-rank Keys for Efficient Sparse Attention (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.