Research map / Vision-language and multimodal / Visual question answering
Falcon Medical Visual Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Falcon Med-VQA, a generative medical VQA system that routes images to pathology or radiology checkpoints of a PMC-CLIP plus Falcon-11B vision-language model, with a UI showing top-3 answers, confidence scores, Grad-CAM maps and GPT-4 explanations.
Problem
Medical visual question answering is usually framed as classification over a fixed answer set, which limits free-form answers; existing generative medical VLMs are limited in accuracy and usability for clinicians.
Approach
Treats Med-VQA as generation using a PMC-CLIP encoder, a query-based transformer connector, and the Falcon-11B LLM, plus a dynamic image-type classifier that routes each query to a domain-specific checkpoint and an explainability layer (confidence scores, NER-based term extraction, Grad-CAM, GPT-4 explanations).
Key results
- On SLAKE, Falcon Med-VQA reaches 86.6% accuracy on open-set and 89.5% on closed-set questions, significantly outperforming MedVInT-TD and PeFoMed.
- Performance is comparable to MedVInT-TD and PeFoMed on radiology and pathology benchmarks, per the authors.
- User requests are processed in an average of 4.9 seconds in the interface.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Falcon Med-VQA instead of MedVInt-TD and PefoMed | Falcon Med-VQA · SLAKE open-set questions | accuracy | Helped |
| Use Falcon Med-VQA instead of MedVInt-TD and PefoMed | Falcon Med-VQA · SLAKE closed-set questions | accuracy | Helped |
Limitations
- Evaluation details and hardware are sparsely reported in the main text.
- Dynamic routing depends on an image-type classifier; errors there propagate to the VQA model.
- Medical report generation is not yet supported (future work).
Experimental setup
Datasets: PMC-VQA v2 SLAKE PathVQA VQA-RAD
Benchmarks: SLAKE VQA-RAD PathVQA
Metrics: accuracy
Baselines: MedVInT-TD PeFoMed GPT-4
Builds on: MedVInT PMC-CLIP Falcon LLM LoRA Grad-CAM
Closest papers by meaning
- Fine-grained Adaptive Visual Prompt for Generative Medical Visual Question Answering (AAAI 2025 · similarity 0.93)
- OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM (CVPR 2024 · similarity 0.91)
- Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine (AAAI 2025 · similarity 0.90)
- Alignment, Mining and Fusion: Representation Alignment with Hard Negative Mining and Selective Knowledge Fusion for Medical Visual Question Answering (CVPR 2025 · similarity 0.90)
- Visual and Domain Knowledge for Professional-level Graph-of-Thought Medical Reasoning (ICML 2025 · similarity 0.90)
- 3D-RAD: A Comprehensive 3D Radiology Med-VQA Dataset with Multi-Temporal Analysis and Diverse Diagnostic Tasks (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.