Research map / Vision-language and multimodal / Visual question answering
Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded
Ramprasaath R. Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin, Shalini Ghosh, Larry Heck, Dhruv Batra, Devi Parikh
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add HINT gradient-importance fine-tuning | UpDn · VQA-CP test | overall accuracy: 39.49 -> 46.73 | Improved |
| add HINT fine-tuning | UpDn · VQAv2 val | overall accuracy: 62.85 -> 63.38 | Improved |
| supervise attention weights (Attn. Align) instead of gradient importance (ablation) | UpDn · VQA-CP test | overall accuracy: 39.49 -> 39.37 | No clear effect |
| supervise attention weights (Attn. Align) (ablation) | UpDn · VQAv2 val | overall accuracy: 62.85 -> 63.24 | Improved |
| use HINT instead of adversarial regularization | UpDn · VQA-CP test | overall accuracy: 41.17 -> 46.73 | Improved |
| add HINT | UpDn · VQA-CP test | Number accuracy: 11.:96 -> 10.61 | Hurt |
| use 1.5% of VQA data with human attention supervision | UpDn + HINT · VQA-CP test | overall accuracy: 5% improvement | Improved |
| add HINT | Bottom-up Top-down captioner · COCO captioning (robust split) | CIDEr: 0.90 -> 0.92 | Improved |
| add HINT | Bottom-up Top-down captioner · COCO captioning (Karpathy split) | CIDEr: 1.06 -> 1.04 | Hurt |
| add HINT | UpDn · VQA-CP | Spearman correlation with human attention: -0.09 -> 0.18 | Improved |
| add HINT | Bottom-up Top-down captioner · COCO robust split | Spearman correlation with segmentation maps: 0.008 -> 0.17 | Improved |
| add HINT | UpDn · VQA-CP | IoU of top proposal with human attention: 0.57 -> 0.63 | Improved |
Closest papers by meaning
- Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding (CVPR 2025 · similarity 0.88)
- In-Context Learning Can Help Vision Language Models Overcome Training Prior (NeurIPS 2026 · similarity 0.88)
- LaViP: Language-Grounded Visual Prompting (AAAI 2024 · similarity 0.87)
- Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning (AAAI 2026 · similarity 0.87)
- Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gain (ICML 2026 · similarity 0.87)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
