Research map / Architectures and efficiency / Attention and transformers
Retrieval-Aware Distillation for Transformer-SSM Hybrids
Aviv Bick, Eric P. Xing, Albert Gu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| retain top 10 retrieval-critical attention heads instead of none (ablation) | Hybrid-Llama-1B · SWDE | accuracy (%): 27.7 -> 71.1 | Improved |
| retain top 10 retrieval-critical attention heads instead of none (ablation) | Hybrid-Llama-1B · KV-Retrieval | accuracy (%): 13.2 -> 99.0 | Improved |
| retain top 10 retrieval heads instead of none (ablation) | Hybrid-Llama-1B · knowledge-focused tasks coverage | coverage vs teacher (%): 101.4 -> 101.3 | No clear effect |
| increase retained attention heads beyond 10-20 (ablation) | Hybrid-Llama-1B · retrieval-heavy coverage | coverage vs teacher (%): 95.0 -> 98.2 at 512 heads | No clear effect |
| place attention by retrieval-importance ranking instead of fixed layer interleaving (ablation) | hybrid distillation · Llama-3.2-1B retrieval-heavy coverage | coverage vs teacher (%): 92.6 (Annealed 25%, 128 heads) -> 95.0 (10 heads) | Improved |
| choose different layers for fixed 25% interleaving (ablation) | Fixed interleaving baseline · Llama-3.2-1B retrieval-heavy coverage | coverage vs teacher (%): 89.3 -> 98.3 | Mixed |
| reduce SSM state dimension from 64 to 8 with 10 retrieval heads (ablation) | Hybrid-Llama-1B · retrieval-heavy coverage | coverage vs teacher (%): 95.0 -> 88.1 | Hurt |
| reduce SSM state dimension from 64 to 8 with 10 retrieval heads (ablation) | Hybrid-Llama-1B · knowledge-focused coverage | coverage vs teacher (%): 101.3 -> 100.8 | No clear effect |
| reduce SSM state dimension further from 8 to 4 (ablation) | Hybrid-Llama-1B · retrieval-heavy coverage | coverage vs teacher (%): 88.1 -> 79.9 | Hurt |
| retain top 10 retrieval heads in Qwen hybrid instead of none (ablation) | Hybrid-Qwen-2.5-1.5B · retrieval-heavy coverage | coverage vs teacher (%): 52.1 -> 96.4 | Improved |
| use retrieval-aware hybrid with fewer heads and d_state 8 instead of heuristic hybrids | inference memory · L=4096 | MB per sequence: 35.8 MB -> 11.0 MB | Improved |
Closest papers by meaning
- Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism (ICML 2025 · similarity 0.90)
- Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling (ICLR 2025 · similarity 0.89)
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.88)
- RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable State (NeurIPS 2026 · similarity 0.87)
- Raven: High-Recall Sequence Modeling via Sparse Memory Routing (NeurIPS 2026 · similarity 0.87)
- Forgetting Transformer: Softmax Attention with a Forget Gate (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
