Research map / Architectures and efficiency / Attention and transformers
On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention for Long-Context LLM Serving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes dual-state linear attention (DSLA) and DSLA-Serve, an on-the-fly adaptive distillation framework that progressively converts Llama2-7B Transformer layers into DSLA layers using sensitivity-based ordering and chained fine-tuning.
Problem
Long-context LLM serving is bottlenecked by quadratic self-attention compute and KV-cache memory, while linear attention alternatives overemphasize recent tokens and static hybrid architectures cannot trade accuracy for efficiency under changing load.
Approach
Each linear-attention layer keeps two hidden states (history and recency gates) blended by a learnable per-layer coefficient, trained with a contrastive regularizer that de-correlates the gates; layers are converted to DSLA in ascending sensitivity order at inference time, with chained fine-tuning so each newly converted layer matches the already-converted configuration.
Key results
- DSLA-Serve yields 2.29x lower normalized end-to-end latency than Llama2-7B on an Azure-trace replay (93.64 ms to 40.83 ms per token).
- DSLA [25%] reaches 56.3% average on lm-eval short-context tasks versus 57.2% for Llama2-7B and 56.9% for Zamba-7B, with 1.6B training tokens.
- Zamba-7B is 3.0x slower than Llama2-7B end-to-end on an A100, while DSLA-Serve is reported 3.0x faster than Zamba-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace self-attention with DSLA in 25% of layers | Llama2-7B · HotpotQA | accuracy: 5.63 -> 11.07 | Helped |
| replace self-attention with DSLA in 25% of layers | Llama2-7B · 2WikiMQA | accuracy: 10.24 -> 14.20 | Helped |
| replace self-attention with DSLA in 25% of layers | Llama2-7B · TriviaQA | accuracy: 86.19 -> 87.46 | Helped |
| replace self-attention with DSLA in 25% of layers | Llama2-7B · WikiText-2 | perplexity: 8.79 -> 9.26 | Hurt |
| replace self-attention with DSLA in 25% of layers | Llama2-7B · CNN/DailyMail | ROUGE-1: 30.13% -> 30.01% | Hurt |
| replace self-attention with DSLA in 25% of layers | Llama2-7B · XSum | ROUGE-L: 24.51% -> 24.70% | Helped |
| replace self-attention with DSLA in 25% of layers | Llama2-7B · lm eval tasks | average score: 57.2% -> 56.3% | Hurt |
| adaptively convert layers with DSLA-Serve | Llama2-7B · end-to-end request serving | latency: 93.64ms -> 40.83ms | Helped |
| add a second specialized hidden state instead of using one state (ablation) | GLA converted from Llama-3.2-1B-Instruct · CoLA validation | accuracy: 78.97 -> 81.19 | Helped |
| use four hidden states instead of two (ablation) | GLA converted from Llama-3.2-1B-Instruct · CoLA validation | accuracy: 81.19 -> 81.44 | No clear effect |
| use eight hidden states instead of two (ablation) | GLA converted from Llama-3.2-1B-Instruct · CoLA validation | accuracy: 81.19 -> 80.96 | Hurt |
Surprising findings
- DSLA [25%] surpasses the pure Transformer teacher on multi-document QA (accuracy gains of 3.96 to 5.44 points).
- Performance begins to degrade when more than 75% of the model is converted to linear attention.
Limitations
- DSLA-Serve keeps both Transformer and DSLA layer weights loaded in memory, adding memory overhead.
- Accuracy degrades noticeably when more than 75% of layers are converted.
- Zamba-7B comparison is measured on A100 rather than the H100 used in its original paper.
Experimental setup
Datasets: SlimPajama C4 CoLA
Benchmarks: WikiText-2 Lambada HotpotQA 2WikiMQA LCC Repobench TREC Samsum TriviaQA CNN/DailyMail XSum WinoGrande
Metrics: perplexity accuracy ROUGE normalized end-to-end latency prefill latency decoding latency
Baselines: Llama2-7B GLA-7B Mamba-7B Zamba-7B RetNet-6.7B LoLCATs Phi-Mamba
Builds on: Gated Linear Attention LoLCATs
Closest papers by meaning
- Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexity (ICLR 2025 · similarity 0.87)
- Viewing Transformers Through the Lens of Long Convolutions Layers (ICML 2024 · similarity 0.87)
- Transformer-Squared: Self-adaptive LLMs (ICLR 2025 · similarity 0.87)
- Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers (NeurIPS 2025 · similarity 0.87)
- The Mamba in the Llama: Distilling and Accelerating Hybrid Models (NeurIPS 2024 · similarity 0.87)
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.