Research map / Vision-language and multimodal / Cross-modal learning
MBQ: Modality-Balanced Quantization for Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Modality-Balanced Quantization (MBQ), which weights vision and language token reconstruction errors by gradient-based sensitivities during calibration for post-training quantization of large VLMs.
Problem
Post-training quantization methods designed for LLMs treat vision and language tokens equally, which over-emphasizes insensitive vision tokens and causes large accuracy loss when quantizing VLMs.
Approach
Measure the sensitivity of the SFT loss to vision versus language token activations via gradients, and use these as modality-balancing weights in the reconstruction loss used to search channel-wise equalization factors, with a MAE-based loss derived from a first-order Taylor expansion.
Key results
- Under W3A16, MBQ improves average accuracy over AWQ by 4.2% on LLaVA-onevision-7B and 18.5% on LLaVA-onevision-72B.
- Under W4A8, MBQ outperforms SmoothQuant and RTN by up to 11.6%.
- The fused W3A16 GEMV kernel achieves 1.4x end-to-end decode speedup on LLaVA-onevision-7B on RTX 4090 (decode 29.6 ms FP16 vs 21.1 ms W3A16).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Balance calibration reconstruction loss by modality sensitivity | LLaVA-onevision-7B W3A16 quantization · MMMU | accuracy: 36.6 -> 42.0 | Helped |
| Balance calibration reconstruction loss by modality sensitivity | LLaVA-onevision-7B W4A8 quantization · MMMU | accuracy: 30.9 -> 42.6 | Helped |
| Balance calibration reconstruction loss by modality sensitivity | LLaVA-onevision-72B W4A8 quantization · MMMU | accuracy: 51.6 -> 55.6 | Helped |
| Use COCO caption calibration data (ablation) | LLaVA-onevision-7B AWQ W3A16 · MMMU | accuracy: 36.6 -> 38.7 | Helped |
| Use COCO caption calibration data (ablation) | LLaVA-onevision-7B SmoothQuant W4A8 · SEED-Bench | accuracy: 41.6 -> 10.2 | Hurt |
| Add modality balancing to calibration (ablation) | LLaVA-onevision-7B SmoothQuant W4A8 with COCO calibration · SEED-Bench | accuracy: 10.2 -> 63.5 | Helped |
| Add modality balancing to calibration (ablation) | LLaVA-onevision-7B AWQ W3A16 with COCO calibration · MMMU | accuracy: 38.7 -> 42.0 | Helped |
| Use MAE rather than MSE for modality-balanced reconstruction (ablation) | LLaVA-onevision-7B W3A16 · MMMU | accuracy: 40.8 -> 42.0 | Helped |
| Use MAE rather than MSE for modality-balanced reconstruction (ablation) | LLaVA-onevision-7B W4A8 · SEED-Bench | accuracy: 63.5 -> 64.4 | Helped |
| Use token-wise gradient reweighting rather than modality-level reweighting (ablation) | LLaVA-onevision-7B W3A16 · OCRBench | accuracy | Hurt |
| Quantize the ViT encoder with W4A8 while quantizing the VLM with MBQ W3A16 (ablation) | LLaVA-onevision-7B · MMMU | accuracy: 42.0 -> 42.6 | Helped |
| Quantize the ViT encoder with W4A8 while quantizing the VLM with MBQ W4A8 (ablation) | LLaVA-onevision-7B · VizWiz | accuracy: 58.9 -> 60.1 | Helped |
Surprising findings
- Directly applying COCO vision-language calibration data to SmoothQuant W4A8 reduces accuracy (SEED drops by 31.4%), while it helps AWQ W3A16.
- AWQ W3A16 on LLaVA-onevision-72B drops about 17% relative to RTN, while MBQ surpasses RTN.
Limitations
- Gains over RTN for weight-activation quantization on large VLMs are limited without rotation, leaving a notable gap to FP16.
- Modality-balancing factors rely on gradients from the SFT loss, requiring backward passes on calibration data.
- Results on W3A16 kernel only shown on RTX 4090 for one model.
Experimental setup
Datasets: ShareGPT4V COCO caption subset (128 samples for calibration) Pile validation (for baseline AWQ/SmoothQuant calibration)
Benchmarks: MMMU SEED-Bench OCRBench VizWiz ScienceQA TextVQA MMLU
Metrics: accuracy average accuracy decode latency speedup kernel speedup
Baselines: RTN GPTQ AWQ SmoothQuant QuaRot SpinQuant
Builds on: AWQ SmoothQuant QuaRot SpinQuant QServe
Closest papers by meaning
- VLM-PTQ: Efficient Post-Training Quantization for Large Vision-Language Models (CVPR 2026 · similarity 0.96)
- Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients (CVPR 2026 · similarity 0.92)
- Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models (NeurIPS 2026 · similarity 0.91)
- Q-VLM: Post-training Quantization for Large Vision-Language Models (NeurIPS 2024 · similarity 0.91)
- Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization (CVPR 2026 · similarity 0.90)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.