Research map / Vision-language and multimodal / CLIP and zero-shot
Beyond Accuracy: On the Effects of Fine-Tuning Towards Vision-Language Model’s Prediction Rationality
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Prediction Trustworthiness and Inference Reliability metrics (using RMA on attention-based heatmaps) to show that common VLM fine-tuning (LP, FLCP, FT) raises correct-but-invalid-evidence predictions while improving reliability given valid evidence.
Problem
Fine-tuning VLMs such as CLIP is usually evaluated only by accuracy, leaving unclear whether fine-tuning affects prediction rationality, i.e., whether correct predictions rest on valid evidence in safety-critical settings.
Approach
Two new metrics built on a 2x2 accuracy-by-rationale grid: PT (fraction of correct predictions with valid evidence) and IR (fraction of valid-evidence samples that are correct), with evidence validity judged by Relevant Mass Accuracy (RMA >= 0.5) on Generic Attention Attribution heatmaps, applied to compare zero-shot and fine-tuned VLMs.
Key results
- On ImageNet-1K with CLIP-ViT-B/16, fine-tuning lowers PT relative to zero-shot by 6.4%, 5.65% and 4.07% for LP, FLCP and FT respectively.
- With CLIP-ViT-B/16 on ImageNet-1K, LP, FLCP and FT improve IR over zero-shot by 12.6%, 8.67% and 16.92%.
- Averaged over datasets and models, zero-shot PT is 71.26% versus 64.78% (LP), 67.95% (FLCP) and 67.01% (FT), while IR averages 56.65% (ZS) rising to 75.67%, 81.71% and 83.52%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune CLIP-ViT-B/16 with LP rather than zero-shot | CLIP-ViT-B/16 · ImageNet-1K | accuracy: 63.30 -> 76.69 | Helped |
| fine-tune CLIP-ViT-B/16 with FLCP rather than zero-shot | CLIP-ViT-B/16 · ImageNet-1K | accuracy: 63.30 -> 72.41 | Helped |
| fine-tune CLIP-ViT-B/16 with FT rather than zero-shot | CLIP-ViT-B/16 · ImageNet-1K | accuracy: 63.30 -> 81.19 | Helped |
| fine-tune CLIP-ViT-B/16 with LP rather than zero-shot | CLIP-ViT-B/16 · ImageNet-1K | PT: 87.05 -> 80.65; $6.4\%$ | Hurt |
| fine-tune CLIP-ViT-B/16 with FLCP rather than zero-shot | CLIP-ViT-B/16 · ImageNet-1K | PT: 87.05 -> 81.40; $5.65\%$ | Hurt |
| fine-tune CLIP-ViT-B/16 with FT rather than zero-shot | CLIP-ViT-B/16 · ImageNet-1K | PT: 87.05 -> 82.98; $4.07\%$ | Hurt |
| fine-tune CLIP-ViT-B/16 with LP rather than zero-shot | CLIP-ViT-B/16 · ImageNet-1K | IR: 66.33 -> 78.93; $12.6\%$ | Helped |
| fine-tune CLIP-ViT-B/16 with FLCP rather than zero-shot | CLIP-ViT-B/16 · ImageNet-1K | IR: 66.33 -> 75.00; $8.67\%$ | Helped |
| fine-tune CLIP-ViT-B/16 with FT rather than zero-shot | CLIP-ViT-B/16 · ImageNet-1K | IR: 66.33 -> 83.25; $16.92\%$ | Helped |
| use AdamW instead of the original optimizer (ablation) | CLIP-ViT-B/32 fine-tuning · ImageNet-1K | PT: 84.23; 85.53; 86.46; ZS 89.39 | Mixed |
| use AdamW instead of the original optimizer (ablation) | CLIP-ViT-B/32 fine-tuning · ImageNet-1K | IR: 74.93; 71.93; 79.26; ZS 61.09 | Helped |
| use gradient-of-attention explanation heatmaps instead of the original heatmap method (ablation) | CLIP-ViT-B/32 fine-tuning · ImageNet-1K | PT: 74.79; 63.62; 65.21; 65.76 | Hurt |
Surprising findings
- Fine-tuning, including FLCP which is claimed to be the standard CLIP fine-tuning method, increases correct predictions based on invalid evidence, lowering prediction trustworthiness.
- Prompt-based or adapter-based tuning (CoOp, Tip-Adapter) also improves IR while lowering PT, so the trade-off persists across fine-tuning families.
Limitations
- Evidence validity is judged by a fixed 0.5 RMA threshold and ground-truth masks from bounding boxes or segmentation, which may not match human notions of valid evidence.
- Relies on attention-based heatmaps (Generic Attention Attribution) and does not evaluate faithfulness of the explanations.
- Results reported from a single run per setting; limited to image classification.
Experimental setup
Datasets: ImageNet-1K CalTech-101 Stanford-Dogs CUB-200-2011 ImageNet-C
Benchmarks: ImageNet-1K CalTech-101 Stanford-Dogs CUB-200-2011 ImageNet-C
Metrics: top-1 accuracy Prediction Trustworthiness (PT) Inference Reliability (IR) Relevant Mass Accuracy (RMA)
Baselines: Zero-Shot Linear-Probing FLCP (FLYP) Fine-tuning CoOp Tip-Adapter
Builds on: CLIP Generic Attention Attribution Relevant Mass Accuracy CoOp Tip-Adapter
Closest papers by meaning
- VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.89)
- What matters when building vision-language models? (NeurIPS 2024 · similarity 0.88)
- Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models (ICML 2024 · similarity 0.88)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.88)
- How Does Vision-Language Adaptation Impact the Safety of Vision Language Models? (ICLR 2025 · similarity 0.88)
- Words or Vision: Do Vision-Language Models Have Blind Faith in Text? (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.