Research map / Vision-language and multimodal / Multimodal LLMs
Attention Prompting on Image for Large Vision-Language Models
Runpeng Yu, Weihao Yu, Xinchao Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| overlay query-guided attention heatmap (API) on image | LLaVA-1.5 · MM-Vet | GPT-eval score: 32.8 -> 36.6 | Improved |
| overlay API heatmap | Gemini · VisWiz | accuracy: 50.28 -> 58.58 | Improved |
| overlay API heatmap | GPT-4V · MM-Vet | GPT-eval score: 67.00 -> 67.70 | No clear effect |
| add 'Let's think step by step' text prompt | LVLMs · VQA benchmarks | accuracy/score: Gemini MM-Vet 59.0 -> 30.6 | Hurt |
| use query-agnostic visual prompts (FGVP, SoM) | LLaVA-1.5 · TextVQA | accuracy: 48.32 -> 18.81 (SoM) | Hurt |
| scale up auxiliary model (CLIP-ViT-B to ViT-L-336) (ablation) | API · MMMU | accuracy: 36.03 -> 37.52 | Improved |
| weaker auxiliary CLIP model (ablation) | API · MME | accuracy: 85.50 -> 83.50 | Hurt |
| add mean filter (kernel 3) to heatmap (ablation) | API · MME | accuracy: 83.70 -> 86.20 | Improved |
| extract LLaVA attention from last two layers instead of layer 20 (ablation) | API (LLaVA auxiliary) · MME | accuracy: 86.34 -> 83.10 | Hurt |
| add reflection via evaluation (judge if highlighted region suffices, else use original image) (ablation) | API (LLaVA) · LLaVA-Bench | GPT-eval score: 74.80 -> 76.10 | Improved |
| add re-emphasize hint that answer lies in highlighted areas (ablation) | API (LLaVA) · LLaVA-Bench | GPT-eval score: 74.80 -> 72.70 | Hurt |
| ensemble API heatmaps from CLIP and LLaVA as two images (ablation) | GPT-4V · LLaVA-Bench | GPT-eval score: 103.60 -> 104.80 | Improved |
Closest papers by meaning
- A-VL: Adaptive Attention for Large Vision-Language Models (AAAI 2025 · similarity 0.90)
- Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want (ICLR 2025 · similarity 0.89)
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.89)
- Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning (ICML 2024 · similarity 0.89)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- SA²VP: Spatially Aligned-and-Adapted Visual Prompt (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
