Research map / Architectures and efficiency / Attention and transformers

Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling

ICLR 2025 · Architectures and efficiency › Attention and transformers · approach: State space and recurrent sequence models · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Samba layer-wise interleaves Mamba, SwiGLU MLP, and sliding window attention to build a linear-time hybrid LM that scales to 3.8B parameters and extrapolates to long contexts.

Problem

Language models need both efficient linear-time processing of very long sequences and precise memory recall, but SSMs struggle with recall and attention is quadratic or extrapolates poorly in perplexity.

Approach

Layer-wise hybrid of Mamba (recurrent compression), sliding window attention (precise recall of recent context), and SwiGLU MLP (factual recall), with the SWA window fixed at 2048 so cost stays linear; pretrained at 4K and extrapolates to 1M perplexity, and fine-tuned for passkey/Phonebook retrieval up to 256K.

Key results

• Samba-3.8B-IT reaches 71.9 MMLU, 87.6 GSM8K and 62.8 HumanEval, beating Phi-3-mini-4K-instruct (68.8, 82.5, 58.5) under the same recipe.
• Samba 1.7B achieves perplexity improvements up to 1M context on Proof-Pile when pretrained at 4K, and reaches 3.73x higher prompt-processing throughput than Llama-3 1.6B at 128K.
• After 500-step passkey fine-tuning at 4K, Samba 1.7B retrieves passkeys perfectly up to 256K, while the SWA-based Mistral 1.6B cannot recall beyond 4K.

Measured effects reported in the paper

Surprising findings

• Samba-NoPE without rotary position embedding still shows exploding perplexity beyond the training length, despite using SWA.
• Using a single key-value head improved validation perplexity for both Llama-2-SWA and Samba at small scale.

Limitations

• Zero-shot passkey retrieval of the pretrained base model is similar to SWA models, so retrieval needs instruction tuning.
• Hybrid strategy is not uniformly better; Mamba-SWA-MLP wins on some tasks such as WinoGrande, SIQA and GSM8K.
• Perplexity-based extrapolation is not matched by zero-shot retrieval; Mamba perplexity still grows slowly at 1M lengths.

Experimental setup

Datasets: Phi-3 pretraining data Phi-2 dataset SlimPajama Proof-Pile Passkey Retrieval (synthetic) Phonebook GovReport SQuALITY

Benchmarks: MMLU GSM8K HumanEval GovReport SQuALITY ARC-Easy ARC-Challenge PIQA WinoGrande SIQA HellaSwag BoolQ

Metrics: accuracy perplexity ROUGE-L pass@1 throughput (tokens/s)

Baselines: Phi-3-mini-4K Llama-3 Mistral Mamba Mamba-SWA-MLP Mamba-MLP Llama-2-SWA Sliding RetNet Sliding GLA SE-Llama-3

Builds on: Mamba FlashAttention 2 Self-Extend Griffin Phi-3

Closest papers by meaning

• Simple linear attention language models balance the recall-throughput tradeoff (ICML 2024 · similarity 0.88)
• LongMamba: Enhancing Mamba's Long-Context Capabilities via Training-Free Receptive Field Enlargement (ICLR 2025 · similarity 0.88)
• Hymba: A Hybrid-head Architecture for Small Language Models (ICLR 2025 · similarity 0.88)
• Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism (ICML 2025 · similarity 0.88)
• RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable State (NeurIPS 2026 · similarity 0.87)
• Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generation (NeurIPS 2025 · similarity 0.87)