Research map / Vision-language and multimodal / Multimodal LLMs

Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Self-Critical Inference (SCI), a logit-level counterfactual decoding framework over textual and visual perturbations that generalizes VCD and scales with the number of counterfactual rounds, plus DRBench, a model-specific adaptive robustness benchmark, showing gains for LLaVA-NeXT-8B and Qwen2-VL-7B.

Problem

LVLMs suffer from language bias (hallucination driven by language priors) and language sensitivity (answers change under semantically equivalent prompt rewrites), and fixed robustness benchmarks fail to capture model-specific vulnerabilities.

Approach

Reinterprets VCD as TIE-style counterfactual reweighting of logits and extends it with multiple visual and textual counterfactual inputs aggregated into a single logit-level prediction (max over textual variants, mean over visual variants), so more counterfactual rounds yield more robustness; DRBench adaptively extracts model-specific bias and sensitivity subsets.

Key results

• On the DRBench BS subset, Qwen2-VL-7B rises from 14.52% (base) to 29.50% with SCI5 and 31.72% with SCI7, versus 22.32% for TIE and 23.12% for VCD.
• On the LLaVA-NeXT BS subset, SCI7 reaches 34.92% overall versus 18.75% for the base model and 27.89% for VCD.
• On real-world datasets, Qwen2-VL-SCI5 achieves 80.98% overall versus 80.58% base, while TIE, VCD and M3ID also gain only marginally.

Surprising findings

• Model-specific failure sets barely overlap: LLaVA-NeXT's BS subset gives 18.75% accuracy for LLaVA-NeXT but 60.31% for Qwen2-VL, and vice versa.
• TIE, VCD and M3ID decrease performance on the Others question type on real-world datasets, while SCI does not.

Limitations

• Inference cost scales with number of counterfactual rounds: vanilla SCI7 is 6.68x base inference time; batching reduces it to 2.48x.
• DRBench samples are derived from the same counterfactual construction used by SCI, so gains may partly reflect overlap; cross-model gains are smaller.
• Gains on standard benchmarks are marginal because vulnerable samples are a small portion of the data.

Experimental setup

Datasets: MME MMStar CCBench ViLP MMBench-DEV-EN-V11 MMBench-DEV-CN-V11

Benchmarks: DRBench (Bias, Sensitivity, BS subsets) MMBench-DEV-EN-V11 MMBench-DEV-CN-V11 MME CCBench MMStar ViLP

Metrics: top-1 accuracy

Baselines: TIE VCD M3ID base model greedy/top-k decoding

Builds on: Visual Contrastive Decoding CF-VQA M3ID Contrastive Decoding

Closest papers by meaning

• MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs (ICML 2026 · similarity 0.91)
• Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning (ICML 2024 · similarity 0.90)
• Enhancing Large Vision Language Models with Self-Training on Image Comprehension (NeurIPS 2024 · similarity 0.90)
• From Head to Tail: Towards Balanced Representation in Large Vision-Language Models through Adaptive Data Calibration (CVPR 2025 · similarity 0.89)
• Diversity Matters: Revisiting Test-Time Compute in Vision-Language Models (ICML 2026 · similarity 0.89)
• Robust-R1: Degradation-Aware Reasoning for Robust Visual Understanding (AAAI 2026 · similarity 0.89)