Research map / Vision-language and multimodal / Multimodal LLMs
Probing Visual Language Priors in VLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ViLP, a 900-triplet out-of-distribution VQA benchmark probing VLM visual language priors, and Image-DPO, a self-improvement method that trains VLMs on good/corrupted image pairs.
Problem
Vision-language models may over-rely on language and visual priors from training data instead of visual reasoning on out-of-distribution images and questions.
Approach
Build a benchmark of generated out-of-distribution images paired with distractor-fact questions and three answers (one prior, two visual-test answers), and train VLMs with an Image-DPO objective that keeps question and answer fixed while corrupting the image (semantic edits, blur, pixelation) to push reliance on visual input.
Key results
- GPT-4o scores 66.17% on ViLP-F Score versus ~98% for human participants.
- Image-DPO raises LLaVA-1.5-7B ViLP-F Score from 29.67 to 34.17 and Cambrian-8B from 8.67 to 20.83.
- Image-DPO outperforms self-improvement baselines (HADPO, RLHF-V, EOS, SIMA, V-DPO) and Text-DPO on ViLP-F Score for LLaVA-1.5-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with Image-DPO instead of the base model | LLaVA-1.5-7B · ViLP F | Score: 29.67 -> 34.17 | Helped |
| Train with Image-DPO instead of the base model | LLaVA-1.5-7B · ViLP P | Score: 37.67 -> 39.33 | Helped |
| Use Image-DPO rather than standard Text-DPO (ablation) | LLaVA-1.5-7B · ViLP F | Score: 31.34 -> 34.17 | Helped |
| Use Text-DPO with corrupted images rather than Image-DPO (ablation) | LLaVA-1.5-7B · ViLP F | Score: 31 -> 34.17 | Hurt |
| Use Text-DPO with corrupted images rather than Image-DPO (ablation) | LLaVA-1.5-7B · ViLP P | Score: 37.5 -> 39.33 | Hurt |
| Use Text-DPO with corrupted images rather than Image-DPO (ablation) | LLaVA-1.5-7B · MM-Vet | score: 30.3 -> 32.3 | Hurt |
| Replace original images with realistic GPT-4o-generated images (ablation) | GPT-4o · ViLP F | Score: -2.2% | Hurt |
| Replace original images with realistic GPT-4o-generated images (ablation) | GPT-4o-mini · ViLP P | Score: -2.2% | Hurt |
| Replace original images with realistic GPT-4o-generated images (ablation) | Claude-Sonnet-3.5 · ViLP F | Score: -4.4% | Hurt |
| Replace original images with realistic GPT-4o-generated images (ablation) | Claude-Opus-3 · ViLP F | Prior: 2.2% | Helped |
| Add distractor facts to questions (ablation) | GPT-4o · ViLP test examples | Score | Helped |
| Add distractor facts to questions (ablation) | LLaVA-1.5-13B · ViLP test examples | Score | Hurt |
Surprising findings
- GPT-4o benefits from distractor facts on the test answers, while weaker models like LLaVA-1.5-13B are misled by them.
- Cambrian-8B and LLaVA-OneVision-72B often fail to follow single-word answer instructions, especially when distractor facts are present.
Limitations
- Benchmark is small (300 questions, 900 triplets) and relies heavily on human curation.
- Generated images are cartoon-like; realistic images in a 45-pair subset slightly reduce scores, so the benchmark's style may affect results.
- Approximately $2.50 cost per QIA triplet and substantial human labor limit scalability.
Experimental setup
Datasets: COCO Text2VQA Visual Genome ViLP
Benchmarks: ViLP NaturalBench MM-Vet CHAIR
Metrics: accuracy (Score) accuracy (Prior) CHAIR-S CHAIR-I
Baselines: GPT-4o Claude-3.5-Sonnet HADPO RLHF-V EOS SIMA V-DPO Text-DPO
Builds on: DPO Self-Rewarding Language Models RLHF-V Winoground HallusionBench
Closest papers by meaning
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.92)
- Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (ICML 2024 · similarity 0.92)
- Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs? (ICML 2025 · similarity 0.91)
- VILA: On Pre-training for Visual Language Models (CVPR 2024 · similarity 0.91)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.91)
- Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMs (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.