Research map / Vision-language and multimodal / Multimodal LLMs

Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Quant Experts (QE), a token-aware PTQ method for VLMs that splits important channels into token-independent (shared low-rank expert) and token-dependent groups (co-occurrence-clustered routed low-rank experts with a router), evaluated on Qwen2VL and InternVL2 at W4A6/W4A8/W3A16.

Problem

Post-training quantization of vision-language models degrades accuracy because important (outlier/sensitive) channels shift across modalities and tokens, while existing methods use static identification and global error compensation.

Approach

Partition important channels by occurrence frequency into token-independent and token-dependent groups; compensate global error with a shared low-rank adapter and cluster token-dependent channels via NPMI co-occurrence spectral clustering into routed low-rank experts selected by a lightweight router per token.

Key results

• Under W4A6 on Qwen2VL-2B, QE averages 58.74 vs 55.92 for LQER and 54.73 for MBQ across 12 benchmarks.
• On Qwen2VL-72B at W4A6, QE reports up to 5.09% average accuracy improvement, approaching FP16 performance.
• QE gives NPU prefill kernel speedups of 3.5x-4.5x over fp16 for Qwen2VL-7B linear layers (analytical performance model).

Surprising findings

• Random routing and random clustering ablations still recover much accuracy, and SE-only results are close to full QE on some metrics.

Limitations

• Routed experts and refinement add memory overhead that grows with the number of routed experts N_r.
• Refinement is optional and the W4A6 gains depend on calibration with only 128 image-caption pairs.

Experimental setup

Datasets: COCO Caption (ShareGPT4V enhanced)

Benchmarks: MMMU OCRBench ScienceQA TextVQA VizWiz AI2D ChartQA DocVQA InfoVQA MMStar MuirBench MMLU

Metrics: accuracy

Baselines: RTN SmoothQuant MBQ LQER AWQ

Builds on: LQER MBQ SmoothQuant AWQ ASER

Closest papers by meaning

• VLM-PTQ: Efficient Post-Training Quantization for Large Vision-Language Models (CVPR 2026 · similarity 0.92)
• Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients (CVPR 2026 · similarity 0.92)
• Zero-Shot Quantization via Weight-Space Arithmetic (NeurIPS 2026 · similarity 0.90)
• MBQ: Modality-Balanced Quantization for Large Vision-Language Models (CVPR 2025 · similarity 0.90)
• Q-VLM: Post-training Quantization for Large Vision-Language Models (NeurIPS 2024 · similarity 0.90)
• ERQ: Error Reduction for Post-Training Quantization of Vision Transformers (ICML 2024 · similarity 0.89)