Research map / Vision-language and multimodal / Multimodal LLMs

VLM-PTQ: Efficient Post-Training Quantization for Large Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VLM-PTQ, an asymmetric weight-compensation PTQ method for VLMs that adds a closed-form correction to the rounding target and a modality-aware Hessian-based importance vector for quantizer scales.

Problem

Directly applying GPTQ/GPTAQ weight-compensation PTQ to vision-language models is suboptimal: RTN ignores residual-induced shifts under the asymmetric objective, and uniform treatment of vision and language input channels causes imbalanced quantization parameters.

Approach

Derives a closed-form correction term that shifts the quantization target by the output residual weighted by inverse-Hessian columns, and builds a modality-aware importance vector by mixing separate vision and language Hessian diagonals with a per-layer coefficient chosen by grid search.

Key results

• On Qwen2.5-VL-7B-Instruct at W3A16, average accuracy rises from 65.0% (GPTAQ) to 71.3% (FP16: 77.2%), with MME-RealWorld English going from 38.4% to 51.3%.
• At W2A16 on Qwen2.5-VL-7B-Instruct, average accuracy improves from 43.1% (GPTAQ) to 48.4%.
• On Qwen2.5-VL-72B-Instruct at W3A16, average accuracy is 76.9% vs 71.2% for GPTAQ (FP16 78.2%); calibration of one layer takes about 1020s versus 921s for GPTAQ.

Surprising findings

• Ablation shows the modality-aware importance vector alone contributes most of the gain (65.0% to 70.4%), while the closed-form correction adds about 1.2 points.

Limitations

• Only the language model is quantized; vision encoder is left in full precision for fair comparison.
• Evaluation limited to a fixed set of eight benchmarks and weight-only or W2A8KV8 settings; accuracy at 2-bit remains far below FP16 for several models.
• Modality-aware awareness coefficient requires a per-layer grid search over candidate values on a calibration batch.

Experimental setup

Datasets: ShareGPT4V COCO Caption (calibration, 128 samples)

Benchmarks: ChartQA DocVQA MME-RealWorld OCRBench ScienceQA SeedBench 2 Plus TextVQA

Metrics: accuracy average accuracy

Baselines: GPTQ GPTAQ

Builds on: GPTQ GPTAQ OBQ

Closest papers by meaning

• MBQ: Modality-Balanced Quantization for Large Vision-Language Models (CVPR 2025 · similarity 0.96)
• Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients (CVPR 2026 · similarity 0.93)
• Q-VLM: Post-training Quantization for Large Vision-Language Models (NeurIPS 2024 · similarity 0.92)
• Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization (CVPR 2026 · similarity 0.92)
• SPHQuant: Efficient extreme low bit weight quantization for Vision-Language Models (NeurIPS 2026 · similarity 0.90)
• Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models (NeurIPS 2026 · similarity 0.90)