Research map / Vision-language and multimodal / Visual question answering
Detecting Any instruction-to-answer interaction relationship:Universal Instruction-to-Answer Navigator for Med-VQA
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Uni-Med, which uses an MLLM to generate visual explanations and "real intent" labels for Med-VQA (IAI-Med VQA dataset), plus task-guided token-level cut-mix and intention-guided attention on a PMC-LLaMA-based model, reaching 87.52% closed and 86.12% overall accuracy on SLAKE.
Problem
Med-VQA models struggle to follow user instructions and to ground answers in visual evidence, especially with diverse, inadequately annotated medical images and limited data.
Approach
Use an MLLM with Universal-Navigator prompts to annotate instructions with "real intent" and answers with core visual explanations, then align explanation tokens to image patches for token-level cut-mix and down-weight non-core instruction tokens in LLM attention.
Key results
- Uni-Med reaches 87.52% closed and 86.12% overall accuracy on SLAKE, exceeding MedVInT-PMC-VQA by 1.22% (closed) and 0.92% (overall).
- On VQA-RAD, Uni-Med achieves 87.22% closed and 74.21% open accuracy, versus MedVInT-PMC-VQA's 86.8% and 73.7%.
- Ablation: removing TC-Mix drops SLAKE overall accuracy from 86.12 to 85.77 and removing IGA to 85.73 (Table 4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add IAI-Med training data | LLaVA-1.5 (7B) · SLAKE | Open accuracy: 72.26 -> 77.32 | Helped |
| Add IAI-Med-pre training data | LLaVA-1.5 (7B) · SLAKE | Open accuracy: 72.26 -> 78.13 (+5.87) | Helped |
| Place visual interpretations before the answer (IAI-Med-pre) | LLaVA-1.5 (7B) · VQA-RAD | Closed accuracy: 68.53 -> 69.48 (+0.95) | Helped |
| Remove UNP from the IAI module (ablation) | Uni-Med · VQA-RAD | Overall accuracy: 81.68 -> 80.5 | Hurt |
| Remove UNP from the IAI module (ablation) | Uni-Med · SLAKE | Overall accuracy: 85.52 -> 83.95 | Hurt |
| Add IAI (ablation) | Uni-Med · VQA-RAD | Overall accuracy: 81.78 -> 82.05 | Helped |
| Add IAI (ablation) | Uni-Med · SLAKE | Overall accuracy: 85.82 -> 86.12 | Helped |
| Add TC-Mix (ablation) | Uni-Med · VQA-RAD | Overall accuracy: 81.86 -> 82.05 | Helped |
| Add TC-Mix (ablation) | Uni-Med · SLAKE | Overall accuracy: 85.77 -> 86.12 | Helped |
| Add IGA (ablation) | Uni-Med · VQA-RAD | Overall accuracy: 81.72 -> 82.05 | Helped |
| Add IGA (ablation) | Uni-Med · SLAKE | Overall accuracy: 85.73 -> 86.12 | Helped |
Surprising findings
- Removing the IAI module only slightly lowers performance relative to the full model (81.78 vs 82.05 overall on VQA-RAD), and the paper notes it still beats the baseline.
Limitations
- Visual explanations are generated by GPT-4V and not validated by clinicians in the paper.
- Gains over the previous SOTA are small (0.92% overall on SLAKE), and improvements are within a few points.
- Experiments only on VQA-RAD and SLAKE; the approach is not tested on larger or more diverse medical benchmarks.
Experimental setup
Datasets: PMC-VQA VQA-RAD SLAKE IAI-Med VQA
Benchmarks: VQA-RAD SLAKE
Metrics: accuracy (closed) accuracy (open) overall accuracy standard deviation GPU throughput
Baselines: MedVInT-PMC-VQA LLaVA-Med LLaVA M3AE MUMC VQAMIX PUBMEDCLIP-MEVF
Builds on: PMC-VQA MedVInT LLaVA-Med CutMix
Closest papers by meaning
- Falcon Medical Visual Question Answering (AAAI 2025 · similarity 0.89)
- Alignment, Mining and Fusion: Representation Alignment with Hard Negative Mining and Selective Knowledge Fusion for Medical Visual Question Answering (CVPR 2025 · similarity 0.88)
- Fine-grained Adaptive Visual Prompt for Generative Medical Visual Question Answering (AAAI 2025 · similarity 0.88)
- DiN: Diffusion Model for Robust Medical VQA with Semantic Noisy Labels (CVPR 2025 · similarity 0.87)
- Visual and Domain Knowledge for Professional-level Graph-of-Thought Medical Reasoning (ICML 2025 · similarity 0.87)
- MIMO: A Medical Vision Language Model with Visual Referring Multimodal Input and Pixel Grounding Multimodal Output (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.