Research map / Vision-language and multimodal / CLIP and zero-shot
Enhancing vision-language models for medical imaging: bridging the 3D gap with innovative slice selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Vote-MI, a one-pass unsupervised VAE-plus-clustering method for selecting representative 2D slices from 3D brain MRI to feed 2D VLMs, and releases the BrainMD dataset with two benchmarks.
Problem
Large vision-language models accept only 2D input, which makes them hard to apply to 3D radiological images such as MRI and CT; slice-by-slice analysis is costly and often inadequate.
Approach
An unsupervised, one-pass representative slice selection (Vote-MI) over patch-wise VAE features that picks diverse, representative 2D slices from 3D volumes to serve as zero-shot or few-shot inputs for VLMs.
Key results
- Vote-MI reaches 59.4% slice-selection accuracy (±4.2) on BrainMD-select versus 28.2% for random selection.
- Vote-MI gives an average 14.6% absolute zero-shot gain over random selection, and 16.6% few-shot gain at budget 8.
- With budget 2, Vote-MI few-shot results beat random selection with eight examples on all tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Vote-MI instead of random example selection | VLM few-shot learning · BrainBench, selection budget 2, W/o cancer | accuracy: 21.3 -> 43.5 | Helped |
| Use Vote-MI instead of random example selection | VLM few-shot learning · BrainBench, selection budget 2, Cancer types | accuracy: 18.3 -> 35.3 | Helped |
| Use Vote-MI instead of random example selection | VLM few-shot learning · BrainBench, selection budget 2, Cancer status | accuracy: 19.6 -> 37.7 | Helped |
| Use Vote-MI instead of random example selection | VLM few-shot learning · BrainBench, selection budget 4, W/o cancer | accuracy: 28.1 -> 50.7 | Helped |
| Use Vote-MI instead of random example selection | VLM few-shot learning · BrainBench, selection budget 4, Cancer types | accuracy: 24.5 -> 43.0 | Helped |
| Use Vote-MI instead of random example selection | VLM few-shot learning · BrainBench, selection budget 4, Cancer status | accuracy: 26.7 -> 42.7 | Helped |
| Use Vote-MI instead of random example selection | VLM few-shot learning · BrainBench, selection budget 8, W/o cancer | accuracy: 30.1 -> 55.3 | Helped |
| Use Vote-MI instead of random example selection | VLM few-shot learning · BrainBench, selection budget 8, Cancer types | accuracy: 28.7 -> 46.0 | Helped |
| Use Vote-MI instead of random example selection | VLM few-shot learning · BrainBench, selection budget 8, Cancer status | accuracy: 30.4 -> 46.7 | Helped |
| Use Vote-MI representative slice selection instead of random selection | zero-shot VLMs · BrainBench, W/o cancer | accuracy: 23.5% absolute gain on average | Helped |
| Use Vote-MI representative slice selection instead of random selection | zero-shot VLMs · BrainBench, Cancer types | accuracy: 14.4% absolute gain on average | Helped |
| Use Vote-MI instead of other slice-selection methods | slice selection · BrainMD-select | accuracy: 52.2 (± 4.9), 53.5 (± 7.0), 47.6 (± 5.4), 28.2 (± 8.6) -> 59.4 (± 4.2) | Helped |
Surprising findings
- Using two Vote-MI-selected examples outperforms eight randomly chosen examples across all tasks.
Limitations
- BrainMD comes from a single site, so generalization to other populations is untested.
- Labels were generated with LLM output and manually reviewed, so some inaccuracies may remain.
- Vote-MI still performs significantly worse than radiologist-selected slices.
Experimental setup
Datasets: BrainMD
Benchmarks: BrainMD-select BrainBench
Metrics: accuracy variance across three runs
Baselines: Uncertainty sampling Core-set K-center Random sampling Radiologist selection
Builds on: Variational Autoencoder core-set selection k-center selection uncertainty sampling
Closest papers by meaning
- Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging (NeurIPS 2025 · similarity 0.90)
- Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generation (NeurIPS 2025 · similarity 0.89)
- OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks (CVPR 2026 · similarity 0.88)
- How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images (ICLR 2026 · similarity 0.88)
- Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding (ICLR 2025 · similarity 0.87)
- LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.