Research map / Vision-language and multimodal / Visual question answering
Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds SpuriVerse, a 1364-question VQA benchmark of 124 spurious correlations sourced from GPT-4o errors and validated with synthetic counterfactual images, evaluates 15 LVLMs, and shows fine-tuning on synthetic spurious examples improves generalization to unseen spurious correlations at some cost to non-spurious accuracy.
Problem
Pretrained large vision-language models may rely on dominant but unreliable spurious correlations in zero-shot VQA, and existing spurious correlation benchmarks focus on narrow, finetuning-based settings rather than general LVLM use.
Approach
A bottom-up pipeline that mines GPT-4o errors on real VQA benchmarks, uses LVLM and human annotation to attribute errors to spurious features, and validates them with Stable Diffusion counterfactual image groups (spurious vs core); then tests whether fine-tuning on diverse spurious examples transfers to unseen spurious correlations.
Key results
- Best closed-source model scores only 37.1% on the anchor set (Qwen-VL-Max), below the 40.7% random baseline; o4-mini reaches 51.45% on spurious groups.
- Fine-tuning Qwen2.5-VL-7B on synthetic spurious groups raises held-out spurious accuracy to 75.6% and anchors to 78.4%, versus 41.9% and 35.2% without fine-tuning.
- Training on spurious groups reduces non-spurious accuracy (Llama-3.2-11B 73.44% to 56.80%), indicating an accuracy-robustness trade-off.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune on synthetic spurious-group samples rather than no fine-tuning | Qwen2.5-VL-7B-Instruct · SpuriVerse held-out anchor test set | accuracy: 35.20 to 78.40 | Helped |
| Fine-tune on synthetic spurious-group samples rather than no fine-tuning | Llama-3.2-11B-vision-instruct · SpuriVerse held-out spurious test set | accuracy: 39.20 to 79.12 | Helped |
| Fine-tune on synthetic spurious-group samples rather than no fine-tuning | Llama-3.2-11B-vision-instruct · SpuriVerse non-spurious test set | accuracy: 73.44 to 56.80 | Hurt |
| Fine-tune on anchor examples rather than no fine-tuning | Llama-3.2-11B-vision-instruct · SpuriVerse held-out anchor test set | accuracy: 41.60 to 59.20 | Helped |
| Fine-tune on non-spurious source-benchmark samples rather than no fine-tuning (ablation) | Qwen2.5-VL-7B-Instruct · SpuriVerse held-out anchor test set | accuracy: 35.20 to 34.40 | Hurt |
| Fine-tune on non-spurious source-benchmark samples rather than no fine-tuning (ablation) | Qwen2.5-VL-7B-Instruct · SpuriVerse held-out spurious test set | accuracy: 41.92 to 38.24 | Hurt |
| Fine-tune on non-spurious source-benchmark samples rather than no fine-tuning (ablation) | Llama-3.2-11B-vision-instruct · SpuriVerse held-out anchor test set | accuracy: 41.60 to 48.80 | Helped |
| Fine-tune on a mixed set rather than spurious samples only (ablation) | Llama-3.2-11B-vision-instruct · SpuriVerse non-spurious test set | accuracy: 56.80 to 64.72 | Helped |
| Fine-tune on a mixed set rather than spurious samples only (ablation) | Llama-3.2-11B-vision-instruct · SpuriVerse held-out anchor test set | accuracy: 80.00 to 71.20 | Hurt |
| Use spurious-aware prompting instead of direct prompting | Qwen2.5-7B · SpuriVerse anchor set | accuracy: 37.90 to 51.61 | Helped |
| Use chain-of-thought prompting instead of direct prompting | Qwen2.5-7B · SpuriVerse anchor set | accuracy: 37.90 to 37.90 | No clear effect |
| Increase the proportion of spurious examples in fine-tuning (ablation) | Llama-3.2-11B-vision-instruct and Qwen2.5-VL-7B-Instruct · SpuriVerse anchor and spurious groups | accuracy | Helped |
Surprising findings
- Chain-of-thought prompting gives insignificant gains on spurious correlation samples, and spurious-aware prompting only moderately helps.
- Fine-tuning on the anchor set alone can reduce non-spurious accuracy for Qwen2.5-VL-7B (from 81.20% to 81.36% for anchor, but non-spurious drops for other setups).
Limitations
- Curation relies on GPT-4o errors and two annotators, introducing curation and annotation bias.
- Only multiple-choice VQA is used, limiting evaluation of open-ended generation.
- Synthetic counterfactual images from Stable Diffusion may not fully match real-world distributions.
Experimental setup
Datasets: AOKVQA SEEDBench SEEDBench2 NaturalBench
Benchmarks: SpuriVerse
Metrics: accuracy accuracy gap (non-spurious minus anchor/spurious)
Baselines: GPT-4o Gemini 2.0 Flash Qwen-VL-Max Llama-3.2-11B-Vision-Instruct Qwen2.5-VL-7B-Instruct MMSpuBench
Builds on: Waterbirds CelebA Just Train Twice Group DRO chain-of-thought prompting
Closest papers by meaning
- RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (ICLR 2024 · similarity 0.90)
- Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognition (ICLR 2025 · similarity 0.89)
- Unveiling the Tapestry of Consistency in Large Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.89)
- NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.