Research map / Architectures and efficiency / Quantization
Quamba: A Post-Training Quantization Recipe for Selective State Space Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Quamba, a static 8-bit per-tensor post-training quantization recipe for Mamba selective SSMs that clips input activations to the selective scan via percentile clipping and quantizes outputs in a Hadamard-transformed outlier-free space.
Problem
Existing transformer quantization methods handle SSM (Mamba) activations poorly: the selective scan input is sensitive to quantization error and the SSM output contains massive outliers, hindering efficient W8A8 deployment on cloud and edge devices.
Approach
Percentile clipping (99.999th) on the SSM input x to tighten its quantization step, combined with a fused Walsh-Hadamard transform on the SSM output y (folded into the output projection) to remove outliers, giving a static, per-tensor INT8 scheme with minimal runtime overhead.
Key results
- Quamba 2.8B W8A8 retains 62.2% average zero-shot accuracy vs 63.1% FP16 (0.9% drop) and reduces Orin Nano generation latency by 1.72x.
- On A5000, Quamba-2.8B generation latency drops from 9.86 ms to 8.12 ms (1.21x) and model size from 5.29 GB to 2.76 GB.
- Quamba-quantized Jamba with LLM.int8 attains 72.9% LAMBADA accuracy vs 74.0% FP16, about a 1.1% drop.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add input percentile clipping to naive W8A8 quantization (ablation) | Mamba 2.8B · Zero-shot tasks, average | accuracy: 45.9% -> 48.5% | Helped |
| Add output Hadamard transform without input percentile clipping (ablation) | Mamba 2.8B · Zero-shot tasks, average | accuracy: 45.9% -> 56.4% | Helped |
| Combine input percentile clipping and output Hadamard transform (ablation) | Mamba 2.8B · Zero-shot tasks, average | accuracy: 45.9% -> 62.2% | Helped |
| Change the input clipping percentile (ablation) | Quamba on Mamba 1.4B and 2.8B · LAMBADA | accuracy: Mamba 1.4B: 60.6% (p=99.9), 62.6% (p=99.99), 60.4% (p=99.999); Mamba 2.8B: 66.5% (p=99.9), 66.3% (p=99.99), 65.5% (p=99. | Mixed |
| Apply clipping optimization to Quamba (ablation) | Mamba 2.8B · Zero-shot tasks, average | accuracy: 62.2% -> 62.5% | Helped |
| Quantize Jamba Mamba blocks with Quamba while using LLM.int8 on self-attention and MoE | Jamba · LAMBADA OpenAI | accuracy: fail -> 72.9% | Helped |
| Use Quamba rather than FP16 for Mamba blocks, with LLM.int8 on self-attention and MoE | Jamba · LAMBADA OpenAI | accuracy: 74.0% -> 72.9% | Hurt |
Surprising findings
- Naive static W8A8 quantization of Mamba collapses, while quantizing SSM input x is more damaging to output than quantizing the output itself once outliers are handled.
- Off-the-shelf LLM.int8 applied jointly to Mamba layers in Jamba fails to produce meaningful accuracy, while Quamba on Mamba layers works.
Limitations
- Relies on symmetric per-tensor quantization; asymmetric and log2 variants were not adopted for hardware/framework reasons.
- Low-bit (W4A4, W2A16) quantization of Mamba via transformer methods was reported to generalize poorly, and sub-8-bit SSM quantization is left unexplored.
- Percentile choice varies by model size (99.999 for 130M, 99.9 for 2.8B), requiring per-model tuning.
Experimental setup
Datasets: Pile WikiText2 LAMBADA HellaSwag PIQA ARC WinoGrande
Benchmarks: LAMBADA HellaSwag PIQA ARC-Easy ARC-Challenge WinoGrande WikiText2 Pile
Metrics: zero-shot accuracy average accuracy perplexity latency (TPOT/TTFT/TTLT) model size
Baselines: Mamba-PTQ Mamba-SmQ Mamba-Had dynamic quantization static quantization Pythia QuaRot Quip#
Builds on: Mamba QuaRot SmoothQuant LLM.int8 OCTAV
Closest papers by meaning
- Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Models (ICML 2025 · similarity 0.94)
- SqueezeLLM: Dense-and-Sparse Quantization (ICML 2024 · similarity 0.89)
- ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models (AAAI 2025 · similarity 0.88)
- Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs (ICML 2024 · similarity 0.88)
- QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR 2024 · similarity 0.88)
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.