AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Retrieval-Aware Distillation for Transformer-SSM Hybrids

Aviv Bick, Eric P. Xing, Albert Gu

ICML 2026 · Architectures and efficiency › Attention and transformers · approach: State space and recurrent sequence models · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
retain top 10 retrieval-critical attention heads instead of none (ablation)Hybrid-Llama-1B · SWDEaccuracy (%): 27.7 -> 71.1Improved
retain top 10 retrieval-critical attention heads instead of none (ablation)Hybrid-Llama-1B · KV-Retrievalaccuracy (%): 13.2 -> 99.0Improved
retain top 10 retrieval heads instead of none (ablation)Hybrid-Llama-1B · knowledge-focused tasks coveragecoverage vs teacher (%): 101.4 -> 101.3No clear effect
increase retained attention heads beyond 10-20 (ablation)Hybrid-Llama-1B · retrieval-heavy coveragecoverage vs teacher (%): 95.0 -> 98.2 at 512 headsNo clear effect
place attention by retrieval-importance ranking instead of fixed layer interleaving (ablation)hybrid distillation · Llama-3.2-1B retrieval-heavy coveragecoverage vs teacher (%): 92.6 (Annealed 25%, 128 heads) -> 95.0 (10 heads)Improved
choose different layers for fixed 25% interleaving (ablation)Fixed interleaving baseline · Llama-3.2-1B retrieval-heavy coveragecoverage vs teacher (%): 89.3 -> 98.3Mixed
reduce SSM state dimension from 64 to 8 with 10 retrieval heads (ablation)Hybrid-Llama-1B · retrieval-heavy coveragecoverage vs teacher (%): 95.0 -> 88.1Hurt
reduce SSM state dimension from 64 to 8 with 10 retrieval heads (ablation)Hybrid-Llama-1B · knowledge-focused coveragecoverage vs teacher (%): 101.3 -> 100.8No clear effect
reduce SSM state dimension further from 8 to 4 (ablation)Hybrid-Llama-1B · retrieval-heavy coveragecoverage vs teacher (%): 88.1 -> 79.9Hurt
retain top 10 retrieval heads in Qwen hybrid instead of none (ablation)Hybrid-Qwen-2.5-1.5B · retrieval-heavy coveragecoverage vs teacher (%): 52.1 -> 96.4Improved
use retrieval-aware hybrid with fewer heads and d_state 8 instead of heuristic hybridsinference memory · L=4096MB per sequence: 35.8 MB -> 11.0 MBImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy