Research map / Vision-language and multimodal / Multimodal LLMs
ViCoR: Estimating Visual Necessity via Counterfactual Residuals for Multimodal Medical Data Selection
Haotian Gan, Yudong Li, Xintian Li, Jiarui Han, Weidong Tang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ViCoR, a data selection method for multimodal medical fine-tuning that scores samples by how visually necessary they are, using counterfactual residuals.
Problem
Medical vision-language models suffer hidden stratification and modality imbalance during fine-tuning, where linguistic priors suppress visually grounded learning and yield weakly grounded rationales.
Approach
Estimates visual necessity of each training sample via counterfactual residuals to select data that enforces visual grounding, rather than scoring samples by generic image-text quality or difficulty (the abstract is truncated before details of the scoring).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace counterfactual residual with joint-gradient score (ablation) | ViCoR · GEMeX-ThinkVG, GMAI-MMBench | accuracy: 42.94 -> 51.56 | Improved |
| add shortcut-dominance weighting (ablation) | ViCoR · GEMeX-ThinkVG, GMAI-MMBench | accuracy: 41.10 -> 51.56 | Improved |
| add residual-direction weighting (ablation) | ViCoR · GEMeX-ThinkVG, MedBLINK | accuracy: 58.69 -> 63.68 | Improved |
| add residual thresholding (ablation) | ViCoR · GEMeX-ThinkVG, CheXbench multi-disease accuracy | accuracy: 45.71 -> 46.84 | Improved |
| add medical supervision calibration (ablation) | ViCoR · GEMeX-ThinkVG, CheXbench multi-disease accuracy | accuracy: 42.90 -> 46.84 | Improved |
| add diversity-aware reranking (ablation) | ViCoR · GEMeX-ThinkVG, CheXbench multi-disease accuracy | accuracy: 42.25 -> 46.84 | Improved |
| rank by residual norm only instead of full ViCoR score (ablation) | ViCoR · GEMeX-ThinkVG, CheXbench multi-disease accuracy | accuracy: 44.92 -> 46.84 | Hurt |
| calibration plus diversity without counterfactual residual scoring (ablation) | ViCoR · GEMeX-ThinkVG, CheXbench multi-disease accuracy | accuracy: 40.82 -> 46.84 | Hurt |
| select 20% subset with ViCoR instead of full-data fine-tuning | Qwen3-VL-4B · CXR-CounterFact-Single cross-benchmark avg | average accuracy: 61.63 -> 65.29 | Improved |
| random 20% subset instead of full data | Qwen3-VL-4B · S-Chain cross-benchmark avg | average accuracy: 62.53 -> 50.40 | Hurt |
| ViCoR selection vs best other 20% selection method | InternVL3.5-4B · OmniMedVQA on GEMeX-ThinkVG | accuracy: 88.35 -> 84.18 | Hurt |
| replace mean-image shortcut control with blank, shuffled or wrong image (ablation) | ViCoR scoring · GEMeX-ThinkVG ranking stability | Spearman rho: 0.9675 to 0.9686 | No clear effect |
Closest papers by meaning
- How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images (ICLR 2026 · similarity 0.87)
- MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representations (CVPR 2025 · similarity 0.87)
- MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization (ICML 2025 · similarity 0.86)
- Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generation (NeurIPS 2025 · similarity 0.86)
- OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM (CVPR 2024 · similarity 0.86)
- Enhancing vision-language models for medical imaging: bridging the 3D gap with innovative slice selection (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
