Research map / Vision-language and multimodal / Visual question answering
Fine-grained Adaptive Visual Prompt for Generative Medical Visual Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FAVP, which generates organ-level visual prompts (masks, boxes, circles, contours) via SAM-Med2D and feeds them with the image into a LoRA-tuned ViT, Q-Former and Vicuna/LLaMA-2 for generative medical VQA.
Problem
Generative medical VQA struggles to precisely localize organ-level regions amid background noise when relying only on language prompts, and full fine-tuning of medical LLMs is parameter-inefficient.
Approach
An Adaptive Visual Prompt Creator uses grid keypoints with SAM-Med2D to produce instance-level region prompts (boxes, circles, masks, contours) that are fed alongside the global image into a LoRA-adapted ViT, a Q-Former-based hierarchical extractor, and an LLM to generate open-ended answers.
Key results
- FAVP reaches 71.9 open-set recall on VQA-RAD, about 11.0 points above LLaVA-Med (64.8).
- On DME, FAVP reports 84.73 accuracy and 97.82 consistency, the best among compared methods.
- FAVP trains about 0.1B parameters (about 1.4% of LLaVA-Med's size) with LoRA plus Q-Former.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add adaptive visual prompts (ablation) | FAVP · VQA-RAD | open-set recall: 67.5 -> 71.9 | Helped |
| add adaptive visual prompts (ablation) | FAVP · SLAKE | open-set recall: 85.1 -> 87.2 | Helped |
| use Circle instead of Mask visual prompts (ablation) | FAVP · VQA-RAD | open-set recall: 70.9 -> 71.9 | Helped |
| use Contour instead of Mask visual prompts (ablation) | FAVP · SLAKE | open-set recall: 86.2 -> 87.2 | Helped |
| increase LLM LoRA rank from 0 to 8 (ablation) | FAVP with Vicuna · VQA-RAD | average QA performance: 76.8 -> 80.1 | Helped |
| increase LLM LoRA rank from 8 to 16 (ablation) | FAVP with Vicuna · VQA-RAD | average QA performance: 80.1 -> 77.5 | Hurt |
| increase LLM LoRA rank from 0 to 8 (ablation) | FAVP with Vicuna · SLAKE | average QA performance: 84.4 -> 87.7 | Helped |
| enable both pretraining stages 1 and 2 (ablation) | FAVP · VQA-RAD | open-set recall: 57.7 -> 71.9 | Helped |
| enable both pretraining stages 1 and 2 (ablation) | FAVP · SLAKE | open-set recall: 79.8 -> 87.2 | Helped |
| use stage 1 without stage 2 (ablation) | FAVP · VQA-RAD | open-set recall: 67.8 -> 57.7 | Hurt |
| use stage 2 without stage 1 (ablation) | FAVP · VQA-RAD | open-set recall: 67.8 -> 55.8 | Hurt |
| use FAVP rather than LLaVA-Med | generative MedVQA · VQA-RAD | open-set recall: 64.8 -> 71.9 | Helped |
Surprising findings
- Stage 1 pretraining alone yields performance slightly lower than the non-pretrained baseline on downstream MedVQA.
Limitations
- Validated only on common modalities (X-ray, CT, MRI), not PET, mammography, or histopathology.
- Weak zero-shot reliability; struggles with complex DME grading.
- Optimal visual prompt type varies by dataset, requiring selection.
Experimental setup
Datasets: ROCO radiology PMC-VQA VQA-RAD SLAKE DME
Benchmarks: VQA-RAD SLAKE DME
Metrics: recall (open) accuracy (closed) overall accuracy consistency
Baselines: LLaVA-Med Med-MoE MedVInT-TE PMC-CLIP M3AE LIMOD
Builds on: LLaVA-Med SAM-Med2D Q-Former LoRA
Closest papers by meaning
- Falcon Medical Visual Question Answering (AAAI 2025 · similarity 0.93)
- OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM (CVPR 2024 · similarity 0.90)
- MIMO: A Medical Vision Language Model with Visual Referring Multimodal Input and Pixel Grounding Multimodal Output (CVPR 2025 · similarity 0.89)
- Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine (AAAI 2025 · similarity 0.89)
- Alignment, Mining and Fusion: Representation Alignment with Hard Negative Mining and Selective Knowledge Fusion for Medical Visual Question Answering (CVPR 2025 · similarity 0.89)
- DiN: Diffusion Model for Robust Medical VQA with Semantic Noisy Labels (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.