Research map / Vision-language and multimodal / Visual question answering
Right this way: Can VLMs Guide Us to See More to Answer Questions?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a Directional Guidance VQA task asking VLMs which way to move the camera when visual information is insufficient, with a human-labeled benchmark from VizWiz and a synthetic crop-perturbation training framework that fine-tunes VLMs with LoRA.
Problem
VLMs answer in one shot even when the image is ill-framed and lacks the information needed; they do not indicate how to reframe the camera to acquire missing visual information, which matters for visually impaired users.
Approach
Generate synthetic ill-framed training data by cropping answerable VizWiz images in directional zones, labeling each perturbation with the reverse direction when the model's answer changes, and fine-tune VLMs with LoRA on this six-way guidance task (leave unchanged, left, right, up, down, none).
Key results
- Fine-tuned LLaVA1.5-13b reaches 0.63 F1 and 0.63 accuracy with perturbation range 0.1-0.9 (no shuffling), while GPT-4o zero-shot reaches 0.56 F1 and 0.60 accuracy.
- Zero-shot models perform poorly on reframing cases: GPT-4o ACC(F) 0.19 and LLaVA1.5-13b ACC(F) 0.41 without fine-tuning.
- Fine-tuned LLaVA1.5-7b improves ACC(F) from 0.33 to 0.44 under the 0.1-0.9 range (without shuffling).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune with synthetic guidance data (0.1-0.9 perturbations, unshuffled options) | InstructBlip-7b · Directional Guidance benchmark | Accuracy: 0.31 -> 0.47 | Helped |
| Fine-tune with synthetic guidance data (0.1-0.9 perturbations, unshuffled options) | LLaVA-1.5-7b · Directional Guidance benchmark | Accuracy: 0.42 -> 0.58 | Helped |
| Fine-tune with synthetic guidance data (0.1-0.9 perturbations, unshuffled options) | LLaVA-1.5-13b · Directional Guidance benchmark | Accuracy: 0.53 -> 0.63 | Helped |
| Increase perturbation range from 0.1-0.3 to 0.5-0.7 (ablation) | LLaVA-1.5-7b synthetic-data fine-tuning · Directional Guidance benchmark | Overall Accuracy: 0.24 -> 0.49 | Helped |
| Use perturbation range 0.1-0.9 rather than narrower perturbation ranges (ablation) | LLaVA-1.5-7b synthetic-data fine-tuning · Directional Guidance benchmark | Overall Accuracy: 0.49 | Helped |
| Use the highest perturbation range, 0.7-0.9 (ablation) | LLaVA-1.5-7b synthetic-data fine-tuning · Directional Guidance benchmark | ACC(F): 0.43 -> 0.40 | Hurt |
| Shuffle option-letter associations (ablation) | Fine-tuned VLMs · Directional Guidance benchmark | Accuracy: 0.48 -> 0.56; 0.58 -> 0.54 | Mixed |
Surprising findings
- Shuffling letter-option assignments in training did not consistently help; it hurt with the 0.3-0.7 range, contrary to the anti-memorization intuition.
- CLIP with a linear probe performs comparably to zero-shot InstructBLIP, suggesting a language model is needed for this task.
Limitations
- Task simplified to four directions plus two categories; ignores zoom, rotation, exposure and multi-direction reframing.
- Small benchmark (821 samples) and preliminary experiments not tuned for maximum performance.
- Reliability not guaranteed; requires cautious deployment in high-risk settings.
Experimental setup
Datasets: VizWiz-VQA VizWiz-grounding
Benchmarks: Directional Guidance benchmark (821 samples)
Metrics: F1 accuracy ACC(F)
Baselines: LLaVA-1.5-7b zero-shot LLaVA-1.5-13b zero-shot InstructBLIP-7b zero-shot GPT-4o (CoT) GPT-4v CLIP+linear probe
Builds on: LLaVA-1.5 InstructBLIP LoRA chain-of-thought
Closest papers by meaning
- Making the v in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (CVPR 2017 · similarity 0.92)
- Roses Are Red, Violets Are Blue... but Should VQA Expect Them To? (CVPR 2021 · similarity 0.91)
- Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation (CVPR 2025 · similarity 0.90)
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.90)
- Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (ICLR 2024 · similarity 0.89)
- NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.