Research map / Vision-language and multimodal / CLIP and zero-shot
RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RaVL, a region-level framework that discovers spurious image-feature/text-attribute correlations in fine-tuned CLIP-style VLMs via clustering regions and scoring them, then mitigates them with a region-aware contrastive loss.
Problem
Fine-tuned vision-language models learn spurious correlations between fine-grained image features and textual attributes, degrading zero-shot performance; existing discovery and mitigation methods work at global image level and for unimodal models, and no subgroup or class labels are available.
Approach
Discover spurious correlations by clustering region-level embeddings (K-Medoids, cosine distance) and ranking clusters with a cluster influence score and cluster performance gap; mitigate by a region-aware contrastive loss that aligns non-spurious regions with class text and pushes spurious regions away from correlated labels.
Key results
- Discovery Precision@10 averaged over 654 settings: RaVL 61.8 vs. best baseline Random 21.2 (Spurious-Aware Detection 20.0), a 191% improvement reported over the closest baseline.
- Mitigation on real-world settings (Stage 1 Precision@10 > 0.8): RaVL image worst-group accuracy 40.8 vs. 31.0 for Standard FT and 37.7 for Upsampled FT.
- In the wild, all 10 evaluated off-the-shelf CLIP/OpenCLIP models on SUN397 showed nonzero cluster performance gaps (mean 9.9, range 5.1 to 14.0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RaVL region-level feature discovery instead of the strongest reported baseline | Fine-tuned VLM spurious-correlation discovery · 654 evaluation settings | Precision@10: 61.8 vs 21.2 | Helped |
| Use RaVL region-level feature discovery instead of Spurious-Aware Detection | Fine-tuned VLM spurious-correlation discovery · 369 evaluation settings, τ_eval=20 | Precision@10: 76.2 vs 25.3 | Helped |
| Use RaVL region-level feature discovery instead of Spurious-Aware Detection | Fine-tuned VLM spurious-correlation discovery · 234 evaluation settings, τ_eval=30 | Precision@10: 84.2 vs 32.1 | Helped |
| Use RaVL region-level feature discovery instead of Spurious-Aware Detection | Fine-tuned VLM spurious-correlation discovery · 168 evaluation settings, τ_eval=40 | Precision@10: 91.1 vs 42.0 | Helped |
| Fine-tune with the region-aware loss instead of the strongest baseline | Fine-tuned VLM spurious-correlation mitigation · Real-world settings, Stage 1 Precision@10>0.6 | Image worst-group accuracy: 39.1 vs 37.8 | Helped |
| Fine-tune with the region-aware loss instead of the strongest baseline | Fine-tuned VLM spurious-correlation mitigation · Real-world settings, Stage 1 Precision@10>0.8 | Region worst-group accuracy: 58.5 vs 52.8 | Helped |
| Omit cluster influence score H_c and rank using unweighted G_c only (ablation) | RaVL discovery, CLIP-RN50 · Real-world settings, τ_eval=10, 20, 30, 40 | Precision@10: 46.0 vs 21.2; 54.8 vs 30.0; 72.4 vs 36.2; 83.3 vs 55.0 | Hurt |
| Omit cluster influence score H_c and rank using weighted G_c only (ablation) | RaVL discovery, CLIP-RN50 · Real-world settings, τ_eval=10, 20, 30, 40 | Precision@10: 46.0 vs 40.9; 54.8 vs 51.7; 72.4 vs 63.8; 83.3 vs 66.7 | Hurt |
Surprising findings
- Distilling Failures, George, and Domino often perform near or below the random baseline in discovering fine-grained spurious features.
- PubMedCLIP-RN50 and PubMedCLIP-ViTB/32 show inverse per-class accuracy trends on cardiomegaly, with the RN50 model favoring the no-cardiomegaly class.
Limitations
- Mitigation performance depends on the accuracy of Stage 1 discovery; gains are larger when discovery precision is high.
- Evaluation relies on synthetic and COCO-based controlled settings where spurious correlations are injected by design, so ground truth is artificial in part.
- Spurious-correlation labels for fine-tuning rely on region proposals and text parsing for class labels, which may be noisy.
Experimental setup
Datasets: MNIST FashionMNIST COCO SUN397 Object-CXR ROCO
Benchmarks: MNIST FashionMNIST COCO SUN397 Object-CXR
Metrics: Precision@K image overall accuracy image worst-group accuracy region overall accuracy region worst-group accuracy cluster performance gap
Baselines: Random Distilling Failures George Domino Spurious-Aware Detection Standard FT Upsampled FT VL-ERM VL-GDRO Spurious-Aware Mitigation
Builds on: CLIP Distilling Failures Domino Group DRO Spurious-Aware Mitigation
Closest papers by meaning
- VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models (ICML 2026 · similarity 0.90)
- Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations? (NeurIPS 2025 · similarity 0.90)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.90)
- Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognition (ICLR 2025 · similarity 0.90)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.89)
- Bridge the Modality and Capability Gaps in Vision-Language Model Selection (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.