Research map / Architectures and efficiency / Quantization
Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Quamba2 is a post-training quantization framework for Mamba1 and Mamba2 SSMs supporting W8A8, W4A8, and W4A16 with sort-and-cluster input quantization, per-state-group B/C quantization, and offline cluster-aware weight reordering.
Problem
Deploying large selective state space models on cloud and edge devices is costly, and existing SSM quantization methods either support few bit-widths or degrade badly at low bit-widths such as W4A8, and ignore generalization on multitask benchmarks.
Approach
Exploit channel order preservation and channel/state persistence in SSMs: sort and cluster the input x channels into head/channel groups with per-group 8-bit scales, quantize input-dependent B and C per state group, and reorder weights offline so the SSM output is compute-invariant, combined with Hadamard fusion and head-to-toe 4/8-bit quantization.
Key results
- Quamba2-8B W4A8 averages 69.1% over six zero-shot tasks vs 70.7% FP16, with 3.05x faster generation (TPOT) and 3.89x less memory on A5000.
- Quamba2 W8A8 Mamba2-8B averages 69.8% vs Quamba 64.8% on zero-shot tasks.
- On MMLU, W4A8 Mamba2-8B drops 5.8 points (41.2% vs 47.0% FP16); the searched W4AX-mixed model recovers to 44.0%, and Quamba2 runs Mamba2-8B on Orin Nano 8G at 13 tokens per second.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Quamba2 W8A8 instead of FP16 | Mamba2-8B on A5000, batch 1 · Mamba2-8B A5000 generation | TPOT (ms): 22.73 -> 12.61 | Helped |
| Use Quamba2 W8A8 instead of Quamba W8A8 | Mamba2-8B on A5000, batch 1 · Mamba2-8B A5000 generation | TPOT (ms): 14.12 -> 12.61 | Helped |
| Use W4A8 instead of W8A8 | Quamba2 Mamba2-8B on A5000 · Mamba2-8B generation | TPOT (ms): 12.61 -> 7.43 | Helped |
| Use W4A8 instead of W8A8 | Quamba2 Mamba2-8B on A5000 · Mamba2-8B prefilling | TTFT (ms): 122.33 -> 140.78 | Hurt |
| Use W4A8 instead of W4A16 | Quamba2 Mamba2-8B · Mamba2-8B MMLU (5-shot) | accuracy: 45.3% -> 41.2% | Hurt |
| Automatically search mixed W4A8/W4A16 blocks rather than use pure W4A8 | Quamba2 Mamba2-8B · MMLU (5-shot) | accuracy: 41.2% -> 44.0% | Helped |
| Automatically search mixed W4A8/W4A16 blocks rather than handcraft the first 14 W4A16 layers | Quamba2 Mamba2-8B · MMLU (5-shot) | accuracy: 43.1% -> 44.0% | Helped |
| Add per-state-group quantization for B and C (ablation) | Quamba2 W4A8 · Mamba2-8B LAMBADA | accuracy: 55.1% -> 60.7% | Helped |
| Add sort-and-cluster for x (ablation) | Quamba2 W4A8 · Mamba2-8B LAMBADA | accuracy: 60.7% -> 68.8% | Helped |
| Add Hadamard transform to W4A8 quantization (ablation) | Quamba2 W4A8 · Mamba2-8B LAMBADA | accuracy: fail -> 53.8% | Helped |
| Add Hadamard transform to W4A16 PerG + GPTQ (ablation) | Quamba2 W4A16 · Mamba2-8B LAMBADA | accuracy: 69.2% -> 71.2% | Helped |
| Add GPTQ to W4A16 PerG quantization without Hadamard (ablation) | Quamba2 W4A16 · Mamba2-8B LAMBADA | accuracy: 64.7% -> 69.2% | Helped |
Surprising findings
- W4A8 Mamba2-8B shows a notable MMLU generalization gap (-5.8%) even though LAMBADA drops only -2.1%.
- Larger SSMs are more resilient to quantizing the embedding and output head, with LAMBADA nearly unchanged at 8B.
Limitations
- Accuracy degradation is non-negligible at W4A8, notably on MMLU (-5.8%).
- W4A8 and W4A16 increase time-to-first-token due to dequantization overhead.
- Mixed-precision search for generalization is only partly solved; the authors expect better search algorithms to be needed.
Experimental setup
Datasets: The Pile (calibration, 512 sentences)
Benchmarks: LAMBADA HellaSwag PIQA ARC-easy ARC-challenge WinoGrande MMLU Natural Questions SquadV2
Metrics: accuracy average accuracy over five runs TPOT (ms) TTFT (ms) model size (GB) energy per request (J)
Baselines: Quamba MambaQuant FP16 QuaRot QServe
Builds on: Quamba MambaQuant GPTQ Mamba2 (structured state space duality) QuaRot
Closest papers by meaning
- Quamba: A Post-Training Quantization Recipe for Selective State Space Models (ICLR 2025 · similarity 0.94)
- GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling (NeurIPS 2026 · similarity 0.88)
- Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge (AAAI 2024 · similarity 0.87)
- ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization (NeurIPS 2025 · similarity 0.87)
- QuEST: Stable Training of LLMs with 1-Bit Weights and Activations (ICML 2025 · similarity 0.86)
- Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.