Research map / Label-efficient and robust learning / OOD and anomaly detection
Less Precise Can Be More Reliable: A Systematic Evaluation of Quantization’s Impact on VLMs Beyond Accuracy
Aymen Bouguerra, Daniel Alfonso Montoya Vasquez, Alexandra Gomez-Villa, Chokri Mraidha, Fabio Arnez
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Quantize with QAT (INT8 LSQ) instead of FP32 inference | CLIP ViT-B/32 (WIT) · CIFAR-100 zero-shot | accuracy (%): 64.47 -> 66.30±0.39 | Improved |
| Quantize to INT8 instead of FP16 for noise robustness | CLIP ViT-B/32 (WIT) TensorRT · CIFAR-100 Gaussian blur | accuracy drop (pp): severe blur: 35.64 -> 30.72 | Improved |
| Apply rotation (Rot+LSQ) rather than simple PTQ for quantization (ablation) | W8A8 quantized VLMs · CounterAnimal | Added vulnerability (%): 3.0 -> -0.2 | Improved |
| Add rotation preprocessing to PTQ (ablation) | PTQ · Failure rate (>5% accuracy drop) | failure rate (%): 62.0% -> 5.6% | Improved |
| Use QAT instead of PTQ (ablation) | quantized VLMs · Failure rate (>5% accuracy drop) | failure rate (%): 44.1% -> 25.0% | Improved |
| Tune the logit scale after quantization on proxy data (ablation) | quantized LAION CLIP models · avg of ImageNet, CIFAR-10/100, SUN397 | ECE (%): 3.1% -> 1.1% | Improved |
| Quantize the text encoder as well as the vision encoder (PTQ) (ablation) | quantized CLIP · calibration, all methods | relative ECE change: +98% ECE | Hurt |
| Quantize models pretrained on noisy LAION data | CLIP LAION models · calibration | relative ECE change: +49% ECE | Hurt |
| Use CC3M proxy data rather than noisy YFCC for logit tuning (ablation) | quantized CLIP · proxy dataset choice | ECE retention | Improved |
| Apply W8A8 quantization | 10 VLMs · synthetic corruptions (CIFAR-10-C) | relative robustness gap change (%): +8.9% | Improved |
| Apply W8A8 quantization | 10 VLMs · natural covariate shift (ImageNet-A/R/V2/Sketch) | relative gap change (%): -0.3 | Hurt |
| Reduce bit-width from W8A8 to W4A8 with simple PTQ (ablation) | quantized VLMs · CounterAnimal | Delta RSG (%): 2.6 -> 12.5 | Hurt |
Closest papers by meaning
- VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models (ICML 2026 · similarity 0.90)
- An Empirical Study Into What Matters for Calibrating Vision-Language Models (ICML 2024 · similarity 0.89)
- Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization (CVPR 2026 · similarity 0.89)
- Zero-Shot Quantization via Weight-Space Arithmetic (NeurIPS 2026 · similarity 0.89)
- VLM-PTQ: Efficient Post-Training Quantization for Large Vision-Language Models (CVPR 2026 · similarity 0.89)
- Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
