Research map / Vision-language and multimodal / Multimodal LLMs
Can We Talk Models Into Seeing the World Differently?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measures the texture vs. shape cue bias in ~19 vision language models and shows that language prompts can steer it toward shape or texture, and also low- vs. high-frequency bias.
Problem
Vision-only models are known to be texture-biased, unlike humans who favor shape; it is unclear how such cue biases behave in LLM-based vision language models and whether language prompts can steer them.
Approach
Applies the texture-shape cue-conflict benchmark to a broad zoo of VLMs in VQA and captioning, compares VLMs to their vision encoders, and steers the shape/texture bias via hand-crafted and LLM-optimized natural-language prompts, plus a new frequency cue-conflict dataset.
Key results
- Most VLMs show shape bias above CLIP ViT-L/14 (~60%) but below humans (96%); e.g., InternVL-Chat 1.1 reaches 68.3% shape bias in VQA.
- Language prompts steer shape bias from roughly 49% to 72% without significantly hurting accuracy; automated prompts exceed hand-crafted ones in increasing texture bias.
- VLM decisions differ from their vision encoders (error consistency up to 78.7%), and LLM-side confidence is almost binary, suggesting the LLM drives the final cue choice.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use an automatically searched prompt to increase LF bias | InternVL-Chat 1.1 · frequency-cue-conflict | LF bias: 34.5 -> 38.6* | Helped |
| use an automatically searched prompt to decrease LF bias | LLaVA-NeXT 7B · frequency-cue-conflict | LF bias: 52.4 -> 48.7* | Helped |
| use a searched prompt to maximize LF bias | LLaVA-NeXT 7B · frequency-cue-conflict | accuracy: 82.83 -> 84.25 | Helped |
| use alternative VQA instructions or CLIP-style options (ablation) | LLaVA-NeXT 7B · cue-conflict VQA | shape bias: 59.2; 59.5; 60.2; 59.4; 59.5 | No clear effect |
| use alternative VQA instructions or CLIP-style options (ablation) | LLaVA-NeXT 7B · cue-conflict VQA | accuracy: 82.58 -> 81.92; 81.33; 80.17; 81.33 | Hurt |
| remove 'Keep your response short' (ablation) | LLaVA-NeXT 7B · cue-conflict image captioning | accuracy: $65.08$ -> $68.25$ | Helped |
| remove 'Keep your response short' (ablation) | LLaVA-NeXT 7B · cue-conflict image captioning | shape bias: $64.0$ -> $63.6$ | No clear effect |
| replace 'Keep your response short' with 'Be precise' (ablation) | LLaVA-NeXT 7B · cue-conflict image captioning | shape bias: $64.0$ -> $67.3$ | Helped |
| replace 'Keep your response short' with 'Be precise' (ablation) | LLaVA-NeXT 7B · cue-conflict image captioning | accuracy: $65.08$ -> $64.50$ | Hurt |
| increase temperature (ablation) | LLaVA-NeXT 7B · cue-conflict VQA and image captioning | shape bias | No clear effect |
| increase temperature (ablation) | LLaVA-NeXT 7B · cue-conflict VQA and image captioning | accuracy | Hurt |
| use a language prompt to steer texture/shape bias | VLMs · texture/shape cue-conflict | accuracy | No clear effect |
Surprising findings
- GPT-4V has surprisingly low accuracy (69.75% VQA) largely due to refusals (~10% of VQA conversations) and shows one of the lowest shape biases (47.9%).
- Steering is limited: prompts shift bias but cannot fully override the model's default cue, similar to limited human prompt steering.
Limitations
- Results may not generalize to radically different VLM architectures; only a subset of models yields logits for analysis.
- GPT-4V refusals and generic captions confound accuracy; prompt choice may bias results.
- Automated prompt search optimized on the cue-conflict test set, so found prompts are a proof of concept rather than generalizable SOTA.
Experimental setup
Datasets: cue-conflict (texture/shape, 1,280 samples) ImageNet 16-class subset
Benchmarks: cue-conflict frequency-cue-conflict
Metrics: shape bias cue accuracy error consistency (Cohen's kappa) generic caption ratio
Baselines: ResNet-50 CLIP ViT-L/14 Gemini Pro Vision 1.0 GPT-4V Qwen-VL Plus/Max
Builds on: cue-conflict benchmark (Geirhos et al.) LLaVA CLIP error consistency (Geirhos et al.)
Closest papers by meaning
- See It from My Perspective: How Language Affects Cultural Bias in Image Understanding (ICLR 2025 · similarity 0.91)
- VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.91)
- What matters when building vision-language models? (NeurIPS 2024 · similarity 0.91)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.90)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.90)
- Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation (ICML 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.