Research map / Vision-language and multimodal / Cross-modal learning
Unsupervised Homography Estimation on Multimodal Image Pair via Alternating Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AltO, an unsupervised homography estimation framework for multimodal image pairs that alternates between a geometry-learning phase and a modality-agnostic representation phase, using Barlow Twins and a spatial extension (Geometry Barlow Twins).
Problem
Unsupervised homography estimation fails when the two input images come from different modalities (large modality gap) in addition to geometric displacement, where prior unsupervised methods collapse or fail.
Approach
Alternates (EM-like) between a Geometry Learning phase that trains the registration network with a spatially-extended Barlow Twins loss and a Modality-Agnostic Representation Learning phase that trains the encoder/projector with standard Barlow Twins, avoiding the trivial-solution collapse of joint training.
Key results
- IHN-1 + AltO reaches MACE 3.06 on Google Map, 1.82 on Google Earth, and 3.11 on Deep NIR, the best among unsupervised methods.
- Baselines UDHN, CAU, and biHomE stay near or above the 23 px no-warp level (e.g., biHomE 24.08 on Google Map), indicating training failure.
- Without alternating optimization, AltO variants fail (MACE ~24-26 for most backbones), showing alternation is key.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use alternating optimization rather than training without alternating (ablation) | AltO with DHN, RAFT, IHN-1, or RHWF-1 · Google Map | MACE: 24.09 -> 6.19; 26.21 -> 3.10; 24.37 -> 3.06; 18.88 -> 3.49 | Helped |
| Use alternating optimization rather than training without alternating or MARL (ablation) | AltO with DHN, RAFT, IHN-1, or RHWF-1 · Google Map | MACE: 24.27 -> 6.19; 25.91 -> 3.10; 23.14 -> 3.06; 24.07 -> 3.49 | Helped |
| Add global average pooling to the projector (ablation) | AltO with DHN, RAFT, IHN-1, or RHWF-1 · Google Map | MACE: 24.07 -> 6.19; 24.07 -> 3.10; 24.01 -> 3.06; 24.08 -> 3.49 | Helped |
| Use Barlow Twins for both geometry and modality losses instead of alternative loss combinations (ablation) | IHN-1 + AltO · Google Map | MACE: 3.73 -> 3.06; 3.36 -> 3.06; 3.56 -> 3.06; 4.71 -> 3.06; 3.4 -> 3.06; 21.47 -> 3.06; 4.95 -> 3.06; 23.99 -> 3.06; 3.74 -> | Helped |
| Use a two-stage encoder rather than a one-stage encoder (ablation) | IHN-1 + AltO, ResNet-34 · Google Map | MACE: 3.82 -> 3.06 | Helped |
| Use a two-stage encoder rather than a three-stage encoder (ablation) | IHN-1 + AltO, ResNet-34 · Google Map | MACE: 9.76 -> 3.06 | Helped |
| Add projector stage 4 to stages 2 and 3 (ablation) | IHN-1 + AltO, ResNet-34 · Google Map | MACE: 3.06 -> 3.07 | No clear effect |
| Use IHN-1 + AltO instead of biHomE | Unsupervised homography estimation · Google Map | MACE: 24.08 -> 3.06 | Helped |
| Use IHN-1 + AltO instead of biHomE | Unsupervised homography estimation · Google Earth | MACE: 23.55 -> 1.82 | Helped |
| Use IHN-1 + AltO instead of biHomE | Unsupervised homography estimation · Deep NIR | MACE: 26.37 -> 3.11 | Helped |
| Use IHN-1 + AltO instead of RHWF-1 supervised | Homography estimation · Google Map | MACE: 0.73 -> 3.06 | Hurt |
| Use IHN-1 + AltO instead of RHWF-1 supervised | Homography estimation · Google Earth | MACE: 1.40 -> 1.82 | Hurt |
Surprising findings
- Without alternating optimization, RHWF-1 with MARL reached 18.88 MACE but other backbones stayed near the no-warp level, so the trivial-solution collapse is architecture-dependent.
- Using global average pooling was necessary: without GAP every backbone failed (~24 MACE).
Limitations
- Performance remains slightly below fully supervised training with the same registration network.
- Training is slower because additional modules must be trained to replace ground-truth labels.
- Requires alternating two phases per mini-batch, which the authors suggest as a motivation for a single-phase future design.
Experimental setup
Datasets: Google Map Google Earth Deep NIR
Benchmarks: Google Map Google Earth Deep NIR
Metrics: MACE
Baselines: UDHN CAU biHomE DHN RAFT IHN-1 RHWF-1
Builds on: Barlow Twins UDHN biHomE IHN RAFT
Closest papers by meaning
- SSHNet: Unsupervised Cross-modal Homography Estimation via Problem Reformulation and Split Optimization (CVPR 2025 · similarity 0.89)
- Noisy One-point Homographies are Surprisingly Good (CVPR 2024 · similarity 0.87)
- Adapting Dense Matching for Homography Estimation with Grid-based Acceleration (CVPR 2025 · similarity 0.86)
- MCNet: Rethinking the Core Ingredients for Accurate and Efficient Homography Estimation (CVPR 2024 · similarity 0.86)
- With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You (NeurIPS 2025 · similarity 0.86)
- Mask-Homo: Pseudo Plane Mask-Guided Unsupervised Multi-Homography Estimation (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.