Research map / Vision-language and multimodal / Visual question answering
Alignment, Mining and Fusion: Representation Alignment with Hard Negative Mining and Selective Knowledge Fusion for Medical Visual Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AMiF, a two-stage medical vision-language pre-training framework that unifies inter- and intra-modality alignment with soft-label distillation and hard negative pair classification, plus a gated cross-attention decoder that fuses answer vocabulary for Med-VQA.
Problem
Med-VQA needs better cross-modal representations: existing Med-VLP alignment lacks a unified scheme across modalities and views, overlooks hard negatives caused by similar medical image-report pairs, and knowledge fusion (e.g., UMLS) may inject irrelevant information.
Approach
Combines global contrastive alignment (inter-modality, multi-view image-image, text-text) trained with soft labels distilled from a BiomedCLIP teacher, local optimal-transport token alignment in a second stage, an explicit hard-negative pair classifier, and a Gated Cross-Attention Module that fuses the answer vocabulary as prior knowledge for generative answering.
Key results
- AMiF reaches 80.49% overall accuracy on VQA-RAD, 86.71% on SLAKE, 65.33% on PathVQA and 83.20% on VQA-2019, outperforming the previous best in Table 1 on all four.
- Compared with its soft-label teacher BiomedCLIP, AMiF improves VQA-RAD by 5.29 points and SLAKE by 1.31 points.
- Ablation shows hard negative mining and soft labels together add roughly 1.4 average points in pre-training (e.g., #7 vs #4 soft labels +1.37%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AMiF instead of the strongest listed baseline | Med-VQA · VQA-RAD | accuracy (Overall): 79.38 -> 80.49 | Helped |
| Use AMiF instead of the strongest listed baseline | Med-VQA · SLAKE | accuracy (Overall): 86.05 -> 86.71 | Helped |
| Use AMiF instead of the strongest listed baseline | Med-VQA · PathVQA | accuracy (Overall): 65.10 -> 65.33 | Helped |
| Use AMiF instead of the strongest listed baseline | Med-VQA · VQA-2019 | accuracy (Overall): 82.13 -> 83.20 | Helped |
| Add global text-text alignment to global inter-modality alignment (ablation) | AMiF pre-training · Average across VQA datasets | accuracy: 74.84 -> 75.28 | Helped |
| Add global multi-view and text-text alignment (ablation) | AMiF pre-training · Average across VQA datasets | accuracy: 75.28 -> 76.06 | Helped |
| Add local-level alignment (ablation) | AMiF pre-training · Average across VQA datasets | accuracy: 76.06 -> 76.96 | Helped |
| Add soft-label global alignment (ablation) | AMiF pre-training · Average across VQA datasets | accuracy: 76.96 -> 77.90 | Helped |
| Add hard negative pair discrimination (ablation) | AMiF pre-training with soft labels · Average across VQA datasets | accuracy: 78.33 -> 78.93 | Helped |
| Use soft labels for global alignment (ablation) | AMiF pre-training · Average across VQA datasets | accuracy: +1.37% | Helped |
| Set local alignment and hard-negative loss weights to 1.0 and 0.2 (ablation) | AMiF pre-training · Average across VQA datasets | accuracy: 78.93 -> 77.17 | Hurt |
| Add AMiF pre-training (ablation) | Med-VQA fine-tuning without knowledge fusion · Average across VQA datasets | accuracy: +3.03% | Helped |
Surprising findings
- Fusing the question vocabulary as prior knowledge gives only a small gain (0.38%), while the answer vocabulary gives 0.35%; the gate adds 1.22% on top of answer fusion.
Limitations
- Pre-training relies on a teacher BiomedCLIP for soft labels, so the method depends on an existing medical CLIP model.
- Large trainable model (326M parameters) and a multi-stage pre-training pipeline with many loss terms and hyperparameters (flexibility weight, loss weights).
Experimental setup
Datasets: MIMIC-CXR ROCO MedICaT VQA-RAD SLAKE PathVQA VQA-2019
Benchmarks: VQA-RAD SLAKE PathVQA VQA-2019
Metrics: accuracy (open, closed, overall)
Baselines: MMQ MEVF-BAN CPRD MMBERT ARL M3AE PubMedCLIP CPCR M2I2 MUMC RAMM PeFoMed
Builds on: CLIP BiomedCLIP MGCA UNITER IPOT
Closest papers by meaning
- DiN: Diffusion Model for Robust Medical VQA with Semantic Noisy Labels (CVPR 2025 · similarity 0.91)
- Falcon Medical Visual Question Answering (AAAI 2025 · similarity 0.90)
- OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM (CVPR 2024 · similarity 0.90)
- Enhancing Medical Large Vision-Language Models via Alignment Distillation (AAAI 2026 · similarity 0.89)
- Fine-grained Adaptive Visual Prompt for Generative Medical Visual Question Answering (AAAI 2025 · similarity 0.89)
- Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.