Research map / Vision-language and multimodal / Human and embodied modeling
Sparse-to-dense Multimodal Image Registration via Multi-Task Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SDME, a multi-task feature extractor predicting sparse-matching and dense-alignment features, and a sparse-to-dense registration pipeline (SM initialization refined by DA) for multimodal homography estimation.
Problem
Multimodal image registration under large motion and appearance differences: sparse feature matching is inaccurate in textureless scenes, while dense direct alignment needs good initialization and is costly.
Approach
Combine SM and DA in a sparse-to-dense pipeline, using one multi-task network (SDME) that outputs both SM descriptors/keypoints and DA single-channel feature maps, trained with MGDA-UB gradient balancing and mutual guidance between the two tasks.
Key results
- On GoogleEarth the method reaches APE 1.38 with 100% success rate, versus 1.82 for IHN and 2.49 for PRISE.
- On VIS-IR-drone the method reaches APE 2.75 (99.88% SR), outperforming IHN (3.00) and SuperPoint+SuperGlue (2.84).
- Ablation on GoogleEarth shows MGDA-UB reduces APE from 2.45 to 1.78 and mutual guidance further to 1.38.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the sparse-to-dense method instead of the strongest reported baseline | Ours versus PRISE · MSCOCO | APE: 0.28 -> 0.53 | Hurt |
| Use the sparse-to-dense method instead of the strongest reported baseline | Ours versus PRISE · MSCOCO* | APE: 0.56 -> 0.80 | Hurt |
| Use the sparse-to-dense method instead of IHN | Ours versus IHN · VIS-NIR | APE: 1.39 -> 1.21 | Helped |
| Use the sparse-to-dense method instead of PRISE | Ours versus PRISE · VIS-NIR* | APE: 10.0 -> 1.58 | Helped |
| Use the sparse-to-dense method instead of ReDFeat | Ours versus ReDFeat · GoogleEarth | APE: 1.82 -> 1.38 | Helped |
| Use the sparse-to-dense method instead of IHN | Ours versus IHN · VIS-IR-drone | APE: 3.00 -> 2.75 | Helped |
| Add MITB to the baseline (ablation) | SDME · GoogleEarth | APE: 2.56 -> 2.45 | Helped |
| Add MITB to the baseline (ablation) | SDME · VIS-IR-drone | APE: 3.45 -> 3.37 | Helped |
| Add MGDA-UB after adding MITB (ablation) | SDME · GoogleEarth | APE: 2.45 -> 1.78 | Helped |
| Add MGDA-UB after adding MITB (ablation) | SDME · VIS-IR-drone | APE: 3.37 -> 3.03 | Helped |
| Add mutual guidance after MITB and MGDA-UB (ablation) | SDME · GoogleEarth | APE: 1.78 -> 1.38 | Helped |
| Add mutual guidance after MITB and MGDA-UB (ablation) | SDME · VIS-IR-drone | APE: 3.03 -> 2.75 | Helped |
Surprising findings
- Plain MNN matching with MAGSAC++ on learned features outperformed sophisticated GNN/Transformer matchers (SuperGlue, LoFTR) because DA drives most of the accuracy gain.
Limitations
- Sparse-to-dense pipeline relies on the SM initialization being roughly correct; failures in SM propagate to DA.
- Evaluation uses synthetically sampled homographies on four datasets, with homography-only (flat-world) assumption.
Experimental setup
Datasets: MSCOCO GoogleEarth VIS-NIR VIS-IR-drone
Benchmarks: MSCOCO VIS-NIR GoogleEarth VIS-IR-drone
Metrics: PE<k percentage APE success rate
Baselines: SIFT R2D2 SFD2 ReDFeat DeepLK PRISE DHN MHN IHN SuperPoint+SuperGlue LoFTR
Builds on: R2D2 DeepLK MGDA ReDFeat MAGSAC++
Closest papers by meaning
- Disentangle-then-Align: Non-Iterative Hybrid Multimodal Image Registration via Cross-Scale Feature Disentanglement (CVPR 2026 · similarity 0.87)
- With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You (NeurIPS 2025 · similarity 0.84)
- Efficient LoFTR: Semi-Dense Local Feature Matching with Sparse-Like Speed (CVPR 2024 · similarity 0.83)
- HomoMatcher: Achieving Dense Feature Matching with Semi-Dense Efficiency by Homography Estimation (AAAI 2025 · similarity 0.83)
- Scalable 3D Registration via Truncated Entry-wise Absolute Residuals (CVPR 2024 · similarity 0.83)
- Diffusion Model for Dense Matching (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.