Research map / Vision-language and multimodal / Cross-modal learning
Towards All-in-One Medical Image Re-Identification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MaMI, a single all-in-one ViT-Base model for medical image re-identification across modalities, using a continuous-modality parameter adapter (ComPA) and inter-image difference alignment with medical foundation models, plus a new MedReID benchmark.
Problem
Medical image re-identification (MedReID) for retrieving a patient's historical images and for privacy protection has been studied only for single modalities, and existing models generalize poorly across modalities and lack medical priors.
Approach
A continuous modality representation built from learnable modality bases generates low-rank, input-specific parameters that adapt a modality-agnostic ViT at runtime, while inter-image feature differences from key-feature-selected local features are aligned to medical foundation models (X-ray, CT, fundus, histopathology) to transfer medical priors.
Key results
- Ours achieves 96.89% CMC-R1 on MIMIC-X and 91.49% on Chest-X, outperforming CLIP† (73.00% on Chest-X) and other baselines across all 11 datasets.
- On the HCC-TACE CT dataset, the full model improves from 80.95% (ComPA only) to 88.09% with medical foundation model alignment.
- Qwen-VL-Max and GPT-4o reach only 76.80% and 62.50% on Chest-X, versus 98.80% reported for the proposed model in the supplementary comparison.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MaMI instead of the strongest listed baseline | MedReID methods · MIMIC-X | CMC-R1: 92.90 -> 96.89 | Helped |
| use MaMI instead of the strongest listed baseline | MedReID methods · Chest-X | CMC-R1: 88.21 -> 91.49 | Helped |
| use MaMI instead of the strongest listed baseline | MedReID methods · CCII (Lung-CT) | CMC-R1: 94.04 -> 95.01 | Helped |
| use MaMI instead of the strongest listed baseline | MedReID methods · KIRC (Ab-CT) | CMC-R1: 57.14 -> 82.68 | Helped |
| add ComPA to the CLIP baseline (ablation) | MaMI on MIMIC-X · MIMIC-X | CMC-R1: 92.29 -> 96.60 | Helped |
| add MFM alignment to ComPA (ablation) | MaMI on HCC-TACE · HCC-TACE | CMC-R1: 80.95 -> 88.09 | Helped |
| add instance-adaptive continuous modality conditioning instead of discrete modality labels (ablation) | MaMI on MIMIC-X · MIMIC-X | CMC-R1: 94.67% -> 96.78% | Helped |
| add the modality codebook (ablation) | MaMI on Chest-X · Chest-X | CMC-R1: 90.12% -> 91.49% | Helped |
| set generated-parameter rank to 8 instead of 16 (ablation) | ComPA on Chest-X · Chest-X | CMC-R1: 91.49% -> 85.56% | Hurt |
| increase generated-parameter rank from 16 to 32 (ablation) | ComPA on Chest-X · Chest-X | CMC-R1: 91.49% -> 91.50% | No clear effect |
| set parameter-generation group number to 128 instead of 64 (ablation) | ComPA on Chest-X · Chest-X | CMC-R1: 91.49% -> 90.52% | Hurt |
| add modality-specific selected key features instead of global features for MFM alignment (ablation) | MaMI on Chest-X · Chest-X | CMC-R1: 88.87% -> 90.02% | Helped |
Surprising findings
- Naively fine-tuning CLIP on multi-modality data can hurt some modalities (X-ray drops from 94.21% single-modality to 92.30% multi-modality).
- Medical foundation models pre-trained on one modality (e.g., RetFound, Med-UniC) transfer poorly to other modalities, and fine-tuned MAE beats fine-tuned MoCoV3 on most datasets.
Limitations
- Evaluation for the cross-modality and application studies relies on a licensed private dataset that is not public.
- Medical foundation model alignment requires a different MFM per modality, so the pipeline depends on the availability of such models.
- Inference latency rises modestly with the ComPA parameter generation.
Experimental setup
Datasets: MIMIC-CXR CCII HCC-TACE EyePACS ODIR LUAD Chest-X KIRC LIHC OASIS2 Mess2 GRAPE
Benchmarks: MIMIC-CXR Chest-X CCII HCC-TACE KIRC LIHC OASIS2 Mess2 ODIR GRAPE LUAD
Metrics: CMC-R1
Baselines: MoCoV3 MAE DINOv2 CLIP TransReID BioMedCLIP RetFound CT-CLIP Med-UniC Packhäuser et al. Ganz et al.
Builds on: CLIP ViT LoRA TransReID MoCo
Closest papers by meaning
- MDReID: Modality-Decoupled Learning for Any-to-Any Multi-Modal Object Re-Identification (NeurIPS 2025 · similarity 0.87)
- Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learning (CVPR 2024 · similarity 0.85)
- Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing (ICML 2025 · similarity 0.84)
- FlexiReID: Adaptive Mixture of Expert for Multi-Modal Person Re-Identification (ICML 2025 · similarity 0.84)
- Multi-Modal Object Re-identification via Sparse Mixture-of-Experts (ICML 2025 · similarity 0.84)
- Multi-modal Medical Diagnosis via Large-small Model Collaboration (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.