Research map / Vision-language and multimodal / CLIP and zero-shot

PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Models

CVPR 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: Test-time adaptation of VLMs · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces PARC, a framework that measures VLM prompt sensitivity using 11 language and vision prompt variations, a calibrated reliability score with guarantees, and calibration against random-guess performance, evaluating 22 VLMs across 7 MC-VQA datasets.

Problem

VLMs are sensitive to prompt changes, but existing studies use only noisy corruptions, rely on multiple non-comparable scores, and cannot compare results across datasets or prompt variations with different random-guess baselines.

Approach

Mirror LLM prompt variation classes (reformulations and semantic changes) in both language and vision, combine accuracy and conformal-prediction certainty into one reliability score with explicit guarantees, and rescale all metrics relative to expected random performance so they are comparable across datasets and prompts.

Key results

• Across 22 VLMs, language negation with antonyms (LS-A) and image exchanges (VS-E) nearly reduce calibrated consistency to random (~0), while reformulations are tolerated better than semantic changes.
• InternVL2-40B is the most reliable and consistent model overall, with InternVL2 family models outperforming other families at matched LLM sizes.
• Reliability and calibrated consistency are positively correlated across models and datasets, and calibration reverses accuracy orderings on MMBench (e.g., LS-N vs. O).

Measured effects reported in the paper

Surprising findings

• Image swaps (VS-S) improved average accuracy and reliability, suggesting model confusion toward spatial arrangement.
• Model family is a better predictor of reliability than model size; InternVL2-2B is on par with LLaVA-1.5 13B.

Limitations

• Framework targets white-box VLMs because certainty requires softmax scores, so black-box models are not directly analyzed.
• Limited to multiple-choice VQA; generative tasks would need extra LLM judges.
• Only one semantic vision variation (exchange) fewer than other variations due to cost of photo-realistic edits.

Experimental setup

Datasets: MMBench MIT-States MIT-Attributes VAW-States VAW-Attributes NYU-Depth V2 Fashionpedia

Benchmarks: MMBench MIT-States MIT-Attributes VAW-States VAW-Attributes NYU-Depth V2 Fashionpedia

Metrics: accuracy certainty consistency reliability calibrated scores

Baselines: LLaVA-1.5 LLaVA-1.6 Qwen-VL CogVLM CogVLM2 InternVL2 Cambrian

Builds on: conformal prediction CompBench MMBench

Closest papers by meaning

• VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.91)
• SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Models (CVPR 2025 · similarity 0.90)
• VP-Bench: A Comprehensive Benchmark for Visual Prompting in Multimodal Large Language Models (AAAI 2026 · similarity 0.90)
• Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (ICML 2024 · similarity 0.90)
• Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.90)
• Amend to Alignment: Decoupled Prompt Tuning for Mitigating Spurious Correlation in Vision-Language Models (ICML 2024 · similarity 0.90)