Research map / Architectures and efficiency / Quantization
MambaQuant: Quantizing the Mamba Family with Variance Aligned Rotation Methods
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MambaQuant, a post-training quantization framework for Mamba vision and language models that combines KLT-enhanced Hadamard rotation (offline) and smooth-fused rotation (online) to equalize channel variances, reaching near-FP accuracy at W8A8.
Problem
Post-training quantization of Mamba models fails with existing methods (e.g., QuaRot loses ~21% on Vim-T† at W8A8) because of channel outliers in gate/output projections and matmul inputs, which the parallel scan amplifies and Hadamard rotation does not equalize across channels.
Approach
Multiply the Hadamard matrix by the KLT eigenvector matrix so that rotated channel variances become uniform (offline), and apply a learned smoothing factor before the online Hadamard rotation for output projection and matmul inputs, fusing the smoothing factors into adjacent weights (smooth-fused rotation).
Key results
- Under W8A8, MambaQuant keeps Vim-T† at 77.8 vs 78.3 FP16 while QuaRot gives 57.4, and Vim-S† at 81.4 vs 81.6 FP16.
- Under W4A8 on Vim-T†, MambaQuant reaches 73.7 vs QuaRot's 48.5 (FP16 78.3).
- On Mamba-2.8b language tasks at W8A8, average accuracy is 62.1 vs 62.2 FP16 and QuaRot's 59.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply MambaQuant W8A8 quantization rather than FP16 | Vision Mamba models · ImageNet (Vim) and UCF-101 (Mamba-ND) | accuracy: FP16 76.1 -> 75.6; 78.3 -> 77.8; 80.5 -> 80.3; 81.6 -> 81.4; 80.3 -> 80.1; 81.7 -> 81.2; 83.0 -> 82.8; 89.6 -> 89.0 | Hurt |
| Use MambaQuant W4A8 instead of QuaRot W4A8 | Vision Mamba models · ImageNet (Vim) and UCF-101 (Mamba-ND) | accuracy: 52.7 -> 72.1; 48.5 -> 73.7; 72 -> 79.4; 74.0 -> 80.4; 72.8 -> 79.8; 80.1 -> 80.4; 82.0 -> 81.9; 86.9 -> 88.4 | Helped |
| Use MambaQuant W8A8 instead of QuaRot W8A8 | Mamba-LLM · Five-dataset average accuracy | accuracy: 48.8 -> 50.0; 51.6 -> 53.8; 56.9 -> 58.3; 59.3 -> 62.1 | Helped |
| Use MambaQuant W4A8 instead of QuaRot W4A8 | Mamba-LLM · Five-dataset average accuracy | accuracy: 43.4 -> 43.9; 40.0 -> 45.8; 53.8 -> 54.3; 58.5 -> 58.5 | Mixed |
| Add KLT-Enhanced rotation instead of direct Hadamard rotation (ablation) | MambaQuant offline rotation · Vim-T†, W8A8 | accuracy: 33.9 -> 47.7 | Helped |
| Add KLT-Enhanced rotation instead of direct Hadamard rotation (ablation) | MambaQuant offline rotation · Mamba-790m, W8A8 | accuracy: 50.8 -> 51.3 | Helped |
| Add KLT-Enhanced rotation instead of direct Hadamard rotation (ablation) | MambaQuant offline rotation · Vim-T†, W4A8 | accuracy: 25.1 -> 38.9 | Helped |
| Add KLT-Enhanced rotation instead of direct Hadamard rotation (ablation) | MambaQuant offline rotation · Mamba-790m, W4A8 | accuracy: 40.2 -> 42.3 | Helped |
| Add Smooth-Fused rotation instead of online Hadamard rotation (ablation) | MambaQuant online rotation · Vim-T†, W8A8 | accuracy: 69.7 -> 77.8 | Helped |
| Add Smooth-Fused rotation instead of online Hadamard rotation (ablation) | MambaQuant online rotation · Mamba-790m, W8A8 | accuracy: 51.8 -> 53.3 | Helped |
| Add Smooth-Fused rotation instead of online Hadamard rotation (ablation) | MambaQuant online rotation · Vim-T†, W4A8 | accuracy: 62.0 -> 73.7 | Helped |
| Add Smooth-Fused rotation instead of online Hadamard rotation (ablation) | MambaQuant online rotation · Mamba-790m, W4A8 | accuracy: 43.0 -> 45.8 | Helped |
Surprising findings
- Vim activations show far more extreme channel outliers than ViT (e.g., out_proj top channel maximum 1371.6 vs 11.6 for ViT), and the paper attributes part of this to PScan amplification.
- Standard Hadamard rotation gives little benefit over RTN at W8A8 for Vim-T† (25.0 to 25.1 on Mamba-790m-related ablation), indicating it does not fix Mamba outliers.
Limitations
- 4-bit weight quantization still shows a noticeable accuracy gap, especially for language models at W4A8.
- The KLT-enhanced rotation cannot be efficiently applied online, so it is restricted to offline transformations.
- Calibration-based KLT may generalize imperfectly to out-of-distribution inputs.
Experimental setup
Datasets: ImageNet UCF-101 HellaSwag
Benchmarks: ImageNet UCF-101 ARC-E ARC-C PIQA Winogrande HellaSwag
Metrics: accuracy
Baselines: RTN GPTQ+RTN SmoothQuant QuaRot
Builds on: QuaRot SmoothQuant Mamba Vision Mamba
Closest papers by meaning
- Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data (NeurIPS 2025 · similarity 0.88)
- The Mamba in the Llama: Distilling and Accelerating Hybrid Models (NeurIPS 2024 · similarity 0.88)
- SpinQuant: LLM Quantization with Learned Rotations (ICLR 2025 · similarity 0.87)
- MambaOut: Do We Really Need Mamba for Vision? (CVPR 2025 · similarity 0.87)
- Vision Mamba Mender (NeurIPS 2024 · similarity 0.87)
- Block-Biased Mamba for Long-Range Sequence Processing (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.