Research map / Vision-language and multimodal / Human and embodied modeling
CLIPSyntel: CLIP and LLM Synergy for Multimodal Question Summarization in Healthcare
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the MMQS dataset of 3,015 medical questions paired with images and CLIPSyntel, a training-free pipeline using CLIP disorder identification, LLM context generation, ImageBind-based sentence filtering, and LLM summarization.
Problem
Medical question summarization has been text-only, ignoring visual information that patients could provide about their symptoms, which can be hard to describe in words.
Approach
Combine CLIP-based zero-shot disorder identification (top-3 candidates refined by an LLM) with LLM-generated symptom/test context, filter that context sentence-by-sentence against the image using ImageBind similarity, and feed the filtered knowledge to an LLM to write a visually aware summary.
Key results
- CLIPSyntel with GPT-3.5 reaches ROUGE-1 0.463, ROUGE-L 0.409 and BERTScore 0.921, slightly above the text-only GPT-3.5 baseline (0.451 / 0.396 / 0.920).
- With LLAMA-2, CLIPSyntel improves ROUGE-1 from 0.188 (text only) to 0.249 and BERTScore from 0.822 to 0.864.
- Human evaluation on GPT-3.5 summaries: clinical score 3.62 for CLIPSyntel vs 3.4 for text-only, with factual recall 0.818 vs 0.748; the annotated gold summary scored 4.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CLIP-derived disorder context (CLIP+GPT-3.5 knowledge) to the query prompt (ablation) | GPT-3.5 summary generation · MMQS | ROUGE-1: 0.451 -> 0.452 | No clear effect |
| add CLIP-derived disorder context without filtration (ablation) | LLAMA-2 · MMQS | ROUGE-1: 0.188 -> 0.189 | No clear effect |
| add CLIP context with ImageBind filtration (full CLIPSyntel) over query-only | GPT-3.5 · MMQS | ROUGE-1: 0.451 -> 0.463 | Helped |
| full CLIPSyntel framework over query-only | LLAMA-2, RedPajama, Vicuna, FLAN-T5, GPT-3.5 · MMQS | ROUGE-1, BLEU, BERTScore: consistent gains across all LLMs | Helped |
| full CLIPSyntel framework over query-only (human eval) | GPT-3.5 · MMQS human evaluation | Clinical-EvalScore, Factual Recall, Omission Rate, MMFCM: 3.4->3.62; 0.748->0.818; 0.235->0.2217; 0.9->1.6 | Helped |
| ImageBind filtering of CLIP-derived context versus unfiltered CLIP context (ablation) | GPT-3.5 · MMQS automatic metrics | ROUGE-1: 0.452 -> 0.463 | Helped |
| use top-3 CLIP candidates with GPT-3.5 selection instead of top-1 CLIP prediction (ablation) | disorder identification module · MMQS disorder identification | accuracy: 84% -> 87% | Helped |
| CLIP-based context with FLAN-T5 and full framework (ablation) | FLAN-T5 · MMQS | ROUGE-1: 0.220 -> 0.220 (Clip+GPT-3.5) | No clear effect |
Surprising findings
- Gains from visual context are small for GPT-3.5 and FLAN-T5 on automatic metrics, with FLAN-T5 CLIPSyntel matching the baseline on ROUGE-1 and ROUGE-2 gains being modest overall.
- CLIP alone with the top-1 disorder reaches 84% accuracy, and adding GPT-3.5 over the top-3 candidates raises it only to 87%.
Limitations
- Restricted to zero-shot prompting; prompt variations not examined.
- Risk of misclassifying medical images, which could produce harmful misinformation.
- Dataset covers only 18 symptoms across four body-region groups and 3,015 samples; images sourced via Bing Image Search.
Experimental setup
Datasets: HealthCareMagic (MedDialog-derived) MMQS
Benchmarks: MMQS
Metrics: ROUGE-1 ROUGE-2 ROUGE-L BLEU-1 BLEU-2 BLEU-3 BLEU-4 BERTScore Clinical Evaluation Score Factual Recall
Baselines: LLM (only textual query) Clip+GPT-3.5
Builds on: CLIP ImageBind MeQSum
Closest papers by meaning
- SMMILE: An expert-driven benchmark for multimodal medical in-context learning (NeurIPS 2025 · similarity 0.88)
- EHRNote-ChatQA: A Benchmark for Evidence-Grounded Multi-Turn Clinical Question Answering over Longitudinal Discharge Summaries (NeurIPS 2026 · similarity 0.87)
- Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning (CVPR 2026 · similarity 0.86)
- Falcon Medical Visual Question Answering (AAAI 2025 · similarity 0.86)
- MIMO: A Medical Vision Language Model with Visual Referring Multimodal Input and Pixel Grounding Multimodal Output (CVPR 2025 · similarity 0.86)
- Medical thinking with multiple images (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.