Research map / Architectures and efficiency / Attention and transformers
Distilling to Hybrid Attention Models via KL-Guided Layer Selection
Yanhong Li, Songlin Yang, Shawn Tan, Mayank Mishra, Rameswar Panda, Jiawei Zhou, Yoon Kim
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select softmax layers by KL-guided greedy addition instead of uniform interleaving | hybrid GDN distillation · RULER, Qwen2.5-3B 25% softmax | RULER: 0.6904 -> 0.8631 | Improved |
| select layers by KL-guided greedy addition instead of Uniform | hybrid GDN distillation · RULER, Llama-3.2-3B 25% softmax | RULER: 0.461 -> 0.7539 | Improved |
| use KL-guided selection instead of the strongest baseline (AR) at 12.5% softmax | hybrid GDN distillation · RULER, Qwen2.5-3B 12.5% softmax | RULER: 0.542 -> 0.662 | Improved |
| use KL-guided selection instead of SMART at larger scales | hybrid GDN distillation · RULER, Qwen2.5-1.5B and 7B | RULER: 1.5B 25%: 0.5098 -> 0.5408; 7B 25%: 0.8158 -> 0.8584 | Improved |
| use stage-2 KL importance instead of stage-1 MSE importance (ablation) | GA selection · RULER, Qwen2.5-3B 25% softmax | RULER: 0.5408 -> 0.8631 | Improved |
| use greedy addition instead of greedy removal (ablation) | KL-based selection · RULER, Llama-3.2-3B 25% softmax | RULER: 0.4950 -> 0.7539 | Improved |
| use greedy addition instead of greedy removal (ablation) | KL-based selection · RULER, Qwen2.5-3B 25% softmax | RULER: 0.8259 -> 0.8631 | Improved |
| average GA and GR rankings instead of GA only (ablation) | KL-based selection · RULER, Llama-3.2-3B 25% softmax | RULER: 0.7539 -> 0.5580 | Hurt |
| use GLA-specific selection instead of GDN-selected layers (ablation) | GLA student · RULER, Qwen2.5-3B 25% softmax | RULER: 0.8407 -> 0.6921 | Hurt |
| add more softmax layers (K from 1 to L-1) (ablation) | hybrid distillation · RULER vs commonsense average | RULER / commonsense | Mixed |
| stop selection pass early (25-40% of stage-2 training) (ablation) | layer selection · RULER | RULER: difference under 0.01 | No clear effect |
| use a larger sliding window in an SWA student | sliding-window distillation · RULER vs commonsense | RULER | Mixed |
Closest papers by meaning
- STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs (NeurIPS 2026 · similarity 0.91)
- Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency (NeurIPS 2025 · similarity 0.88)
- SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking (NeurIPS 2026 · similarity 0.87)
- On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention for Long-Context LLM Serving (ICML 2025 · similarity 0.87)
- The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry (ICLR 2024 · similarity 0.87)
- No Head Left Behind – Multi-Head Alignment Distillation for Transformers (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
