Research map / Vision-language and multimodal / Cross-modal learning

Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Contrastive vision-language alignment · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SplitQ, a channel-splitting post-training quantization framework for VLMs that decouples modality-specific outlier channels and adds learnable low-rank cross-modal calibration branches.

Problem

Low-bit PTQ of vision-language models degrades accuracy because text and vision activations have heterogeneous, channel-dependent outlier distributions that a shared transformation cannot handle.

Approach

Split input channels into modality-compatible, text-specific, and vision-specific sets (MOCD) with separate transforms, then add a cross-modal weight-smoothing low-rank branch and a text-only activation-compensation low-rank branch (ACC) built on a constrained SVD-anchored parameterization with learnable diagonal gating.

Key results

• On Qwen2.5-VL-7B at W4A4, SplitQ averages 73.5 vs 74.3 FP16, while MBQ and MASQuant collapse to about 5 and 8 average.
• At W3A3 on Qwen2.5-VL-7B, SplitQ reaches 69.5 average (vs 74.3 FP16); baselines fail to produce normal outputs.
• SplitQ achieves 2.89x prefill speedup and 1.82x memory reduction over FP16 on Qwen2.5-VL-7B (batch size 8, RTX 4090, W4A4).

Surprising findings

• Fixed SVD-based low-rank branches perform worse than the baseline without CWS/MAC, while learnable constrained low-rank branches help.
• Text-only activation compensation captures most of the gain; compensating vision activations too gives negligible additional benefit.

Limitations

• Multi-branch computations add latency and memory overhead relative to simpler PTQ baselines.
• The modality-specific channel ratio is fixed rather than adaptive across architectures and data.
• Only the LLM component is quantized in main experiments; vision encoder quantization is explored only in an appendix.

Experimental setup

Datasets: calibration samples (128 default)

Benchmarks: MMMU SEED-Bench OCRBench VizWiz ScienceQA TextVQA

Metrics: accuracy average accuracy prefill latency speedup memory usage

Baselines: SmoothQuant MBQ MASQuant Q-VLM DuQuant QSVD FlatQuant

Builds on: FlatQuant SmoothQuant SpinQuant OmniQuant LoRA

Closest papers by meaning

• Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients (CVPR 2026 · similarity 0.92)
• MBQ: Modality-Balanced Quantization for Large Vision-Language Models (CVPR 2025 · similarity 0.91)
• VLM-PTQ: Efficient Post-Training Quantization for Large Vision-Language Models (CVPR 2026 · similarity 0.90)
• Q-VLM: Post-training Quantization for Large Vision-Language Models (NeurIPS 2024 · similarity 0.90)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.89)
• LoMo: Local Modality Substitution for Deeper Vision-Language Fusion (NeurIPS 2026 · similarity 0.89)