Research map / Architectures and efficiency / Attention and transformers

Hymba: A Hybrid-head Architecture for Small Language Models

ICLR 2025 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Hymba, a family of sub-2B small language models whose layers fuse parallel attention and SSM (Mamba) heads, plus learnable meta tokens and KV-cache optimizations, achieving SOTA small-LM accuracy with much smaller cache and higher throughput.

Problem

Small language models face a tradeoff: attention gives high-resolution recall but large KV cache and slow inference, while SSMs are efficient but weak at memory recall; sequential hybrids can create information bottlenecks.

Approach

Fuse attention heads and SSM heads in parallel within the same layer (normalized, re-scaled, averaged), add learnable meta tokens prepended to all inputs that act as learned cache initialization, and reduce the KV cache via mostly sliding-window attention with a few global layers and cross-layer KV sharing.

Key results

• Hymba-1.5B achieves 61.06% average on the commonsense/QA suite, best among sub-2B models, and 1.32% higher average than Llama-3.2-3B with 11.67x smaller cache and 3.49x throughput.
• Compared with SmolLM2-1.7B (11T tokens), Hymba-1.5B (1.5T tokens) gets +1.02% average accuracy, 19.91x smaller cache, and 2.79x throughput.
• Hymba-1.5B-Instruct reaches 49.22% average across MMLU, IFEval, GSM8K, GPQA, and BFCLv2, ahead of Qwen2.5-1.5B-Instruct (47.30%).

Measured effects reported in the paper

Surprising findings

• Applying the same KV cache optimizations (sliding window, KV sharing) to a pure Transformer drops recall accuracy by 10% or more, whereas the hybrid model retains recall.
• Replacing all global attention with sliding window attention in the hybrid collapses recall (accuracy dropping over 20% on recall-intensive tasks), but restoring global attention in just three layers recovers it.

Limitations

• Pretraining uses a proprietary high-quality dataset; the public-data-only variant drops on 5-shot MMLU (44.31 vs 51.19).
• Head-importance and ablation results are reported on a single 300M/1B setting and a few tasks.

Experimental setup

Datasets: DCLM-Baseline-1.0 SmolLM-Corpus FineWeb RoleBench SQuAD GSM8K

Benchmarks: MMLU ARC-C ARC-E PIQA HellaSwag Winogrande SQuAD-C SWDE GSM8K GPQA IFEval BFCLv2

Metrics: accuracy throughput (tokens/sec) cache size (MB) perplexity

Baselines: Llama-3.2-1B Llama-3.2-3B SmolLM2-1.7B Qwen2.5-1.5B Phi-1.5 h2o-danube2-1.8B Rene-v0.1 Samba

Builds on: Mamba Mamba-2 Samba DoRA DPO

Closest papers by meaning

• Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling (ICLR 2025 · similarity 0.88)
• DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads (ICLR 2025 · similarity 0.88)
• Simple linear attention language models balance the recall-throughput tradeoff (ICML 2024 · similarity 0.88)
• CHAI: Clustered Head Attention for Efficient LLM Inference (ICML 2024 · similarity 0.87)
• CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration (ICML 2025 · similarity 0.87)
• Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism (ICML 2025 · similarity 0.86)