Research map / Image generation / Autoregressive and latent generation
MINIMA: Modality Invariant Image Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MINIMA, a data engine that uses generative models to scale RGB matching data (MegaDepth) into the multimodal MD-syn dataset, then fine-tunes LightGlue, LoFTR, and RoMa on it for cross-modal image matching.
Problem
Image matching across different imaging modalities (RGB-IR, RGB-Depth, RGB-Event, etc.) suffers from modality gaps and scarce, small, imbalanced multimodal training data, so existing matchers generalize poorly.
Approach
Generate multiple pseudo-modalities from existing RGB multi-view data (MegaDepth) using generative and simulation models (StyleBooth fine-tuned for infrared, Depth Anything V2, DSINE, event simulation, artistic styles), so that the RGB matching labels are inherited; then pre-train on RGB and fine-tune any matcher on randomly chosen modality pairs to obtain a single unified model.
Key results
- On real RGB-IR METU-VisTIR, MINIMARoMa reaches 37.45 AUC@5° versus 25.61 for RoMa, and MINIMALG reaches 19.14 versus 2.17 for LightGlue.
- On real RGB-Depth DIODE, MINIMARoMa reaches 28.98 AUC@3px versus 9.21 for RoMa, and MINIMALoFTR improves LoFTR from 4.20 to 18.65 AUC@5px.
- On zero-shot Remote Sensing, MINIMARoMa reaches 32.55 AUC@3px versus 29.24 for RoMa.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune RoMa on MD-syn | RoMa · METU-VisTIR | AUC@10°: 48.12 -> 60.70 | Helped |
| fine-tune RoMa on MD-syn | RoMa · DIODE | AUC@5px: 24.64 -> 50.88 | Helped |
| fine-tune RoMa on MD-syn | RoMa · Medical | AUC@10px: 53.75 -> 57.55 | Helped |
| fine-tune RoMa on MD-syn | RoMa · Remote Sensing | AUC@10px: 57.84 -> 64.38 | Helped |
| fine-tune RoMa on MD-syn | RoMa · RGB-Event | AUC@10px: 10.71 -> 17.07 | Helped |
| train LoFTR from scratch on synthetic RGB-IR (ablation) | LoFTR · Synthetic RGB-IR | AUC@10°: 12.58 -> 23.63 | Helped |
| fine-tune LoFTR on real RGB-IR (ablation) | LoFTR · Real RGB-IR | AUC@10°: 6.94 -> 9.78 | Helped |
| fine-tune LoFTR on real RGB-IR (ablation) | LoFTR · Synthetic RGB-IR | AUC@10°: 12.58 -> 6.28 | Hurt |
| fine-tune LoFTR on synthetic RGB-IR (ablation) | LoFTR · Synthetic RGB-IR | AUC@10°: 12.58 -> 29.43 | Helped |
| fine-tune LoFTR on synthetic RGB-Depth (ablation) | LoFTR · Synthetic RGB-IR | AUC@10°: 12.58 -> 17.30 | Helped |
| fine-tune LoFTR on synthetic RGB-Depth (ablation) | LoFTR · Real RGB-Depth | AUC@10px: 15.16 -> 36.06 | Helped |
| fine-tune LoFTR on mixed synthetic RGB-IR, RGB-Depth, and RGB-Normal (ablation) | LoFTR · Synthetic RGB-IR | AUC@10°: 29.43 -> 32.36 | Helped |
Surprising findings
- The original LoFTR and ELoFTR underperform SuperGlue and LightGlue on the synthetic MD-syn benchmark, and XoFTR is competitive despite being a modality-specific model.
- Fine-tuning on synthetic RGB-Depth alone generalizes to real RGB-IR better than fine-tuning on real RGB-IR data (LLVIP and M3FD).
Limitations
- Gap between real and pseudo modalities, and fake information introduced by generative models, are not fully quantified (discussed only qualitatively)
- Semi-dense MINIMALoFTR trails XoFTR on RGB-IR and RGB-Depth, since the base LoFTR is an older architecture
- Generating 480M pairs with diffusion-based generative models is computationally expensive
Experimental setup
Datasets: MegaDepth LLVIP M3FD DIODE METU-VisTIR DSEC MMIM
Benchmarks: MD-syn METU-VisTIR DIODE DSEC MMIM Medical Remote Sensing
Metrics: AUC of pose error AUC of homography corner error runtime (ms) pairs per second
Baselines: SuperGlue LightGlue ReDFeat GIM OmniGlue LoFTR ELoFTR XoFTR DKM RoMa
Builds on: LightGlue LoFTR RoMa StyleBooth Depth Anything V2
Closest papers by meaning
- Self-Supervised Spatial Correspondence Across Modalities (CVPR 2025 · similarity 0.82)
- Robust Noisy Correspondence Learning with Equivariant Similarity Consistency (CVPR 2024 · similarity 0.80)
- MDReID: Modality-Decoupled Learning for Any-to-Any Multi-Modal Object Re-Identification (NeurIPS 2025 · similarity 0.80)
- Latent Modulated Function for Computational Optimal Continuous Image Representation (CVPR 2024 · similarity 0.80)
- Image-to-Image Matching via Foundation Models: A New Perspective for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.80)
- Multi-Modal Object Re-identification via Sparse Mixture-of-Experts (ICML 2025 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.