Research map / Vision-language and multimodal / Cross-modal learning
Self-Supervised Spatial Correspondence Across Modalities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends contrastive random walks with intra- and cross-modal cycle consistency to learn dense pixel correspondences across RGB, depth, thermal, sketch, and stylized images from unlabeled data.
Problem
Finding pixel-level space-time correspondences between images from different visual modalities (e.g., RGB-depth, RGB-thermal, photo-sketch, cross-style) without paired multimodal data or photo-consistency assumptions.
Approach
Extend the contrastive random walk with a global matching transformer to cross-modal cycles, adding auxiliary intra-modal cycles on augmented crops and an edge-aware smoothness loss so the learned embedding supports both intra- and cross-modal matching without translation or photometric assumptions.
Key results
- On RGB-to-depth/thermal cross-modal matching, the method reaches <δx avg of 34.3 (NYU D->RGB... RGB->D 33.5) versus at most 16.6 for the best unsupervised baseline on NYU and 47.9 on Thermal-IM T->RGB versus 8.2 for the best prior baseline.
- On PSC6K photo-sketch matching, the DINO-based variant with all stages reaches PCK-5 53.61 and PCK-10 82.20, competitive with PSCNet (57.92 / 84.72, supervised on PSC6K).
- On cross-style matching, the method reaches PCK-5 69.26 and PCK-10 84.19, outperforming GeoAwareSC (68.30 / 79.67) and SD-DINO (60.14 / 80.72).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add intra-modal random-walk losses before joint training with cross-modal losses (ablation) | Self-Supervised Spatial Correspondence Across Modalities · NYU Depth V2, RGB→Depth | < δx avg: 5.6 -> 19.1 | Helped |
| add intra-modal random-walk losses before joint training with cross-modal losses (ablation) | Self-Supervised Spatial Correspondence Across Modalities · Thermal-IM, RGB→Thermal | < δx avg: 6.2 -> 30.2 | Helped |
| add spatial smoothness loss (ablation) | Self-Supervised Spatial Correspondence Across Modalities · NYU Depth V2, RGB→Depth | < δx avg: 19.1 -> 33.5 | Helped |
| add spatial smoothness loss (ablation) | Self-Supervised Spatial Correspondence Across Modalities · Thermal-IM, RGB→Thermal | < δx avg: 30.2 -> 41.8 | Helped |
| use only cross-modal losses instead of intra-modal pretraining followed by joint training (ablation) | Self-Supervised Spatial Correspondence Across Modalities · NYU Depth V2, RGB→Depth | < δx avg: 19.1 -> 5.6 | Hurt |
| replace the CNN encoder with pretrained DINOv2 and fine-tune it (ablation) | Self-Supervised Spatial Correspondence Across Modalities · PSC6K | PCK-5: 26.22 -> 50.66 | Helped |
| add Stage 1 intra-modal training to the DINOv2 model (ablation) | Ours (DINO) on PSC6K · PSC6K | PCK-5: 50.66 -> 53.61 | Helped |
| use the learned correspondence framework instead of nearest-neighbor matching on DINOv2 features | Self-Supervised Spatial Correspondence Across Modalities · Cross-style image matching | PCK-5: 23.13 -> 69.26 | Helped |
| use the proposed method instead of the strongest reported baseline | Self-Supervised Spatial Correspondence Across Modalities · NYU-Depth V2, RGB→Depth | < δx avg: 16.2 -> 33.5 | Helped |
| use the proposed method instead of the strongest reported baseline | Self-Supervised Spatial Correspondence Across Modalities · Thermal-IM, RGB→Thermal | < δx avg: 29.3 -> 41.8 | Helped |
| use the proposed method instead of GeoAwareSC | Self-Supervised Spatial Correspondence Across Modalities · Cross-style image matching | PCK-5: 68.30 -> 69.26 | Helped |
Surprising findings
- Directly optimizing cross-modal cycle consistency plateaus at a poor solution; intra-modal pretraining is needed for convergence.
- Supervised optical flow models (RAFT, GMFlow) applied to depth/thermal inputs perform far worse than the proposed self-supervised method on cross-modal matching.
Limitations
- Evaluated on only four domains (RGB-depth, RGB-thermal, photo-sketch, cross-style), and all include RGB images.
- Occasional failures in cross-style matching, such as confusing left and right limbs of animals.
- Relies on occluding contours and distinctive visual structure that may not exist in other modalities.
Experimental setup
Datasets: NYU Depth V2 Thermal-IM KAIST PSC6K ImageNet (prompts via BLIP-2)
Benchmarks: NYU Depth V2 (RGB-Depth matching) Thermal-IM KAIST PSC6K Cross-style image matching benchmark
Metrics: <δx avg (positional accuracy) PCK-5 PCK-10
Baselines: RAFT GMFlow Arar et al. CycleGAN+GMFlow ARFlow DIFT SD-DINO DINOv2+NN GeoAwareSC PSCNet WarpC-Net
Builds on: contrastive random walk Global Matching Random Walk (GMRW) DINOv2 ARFlow TAP-Vid
Closest papers by meaning
- CrossOver: 3D Scene Cross-Modal Alignment (CVPR 2025 · similarity 0.87)
- Self-Supervised Cross-View Correspondence with Predictive Cycle Consistency (CVPR 2025 · similarity 0.86)
- C3Po: Cross-View Cross-Modality Correspondence by Pointmap Prediction (NeurIPS 2025 · similarity 0.86)
- SemAlign3D: Semantic Correspondence between RGB-Images through Aligning 3D Object-Class Representations (CVPR 2025 · similarity 0.86)
- Noisy Correspondence Rectification via Asymmetric Similarity Learning (AAAI 2025 · similarity 0.85)
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.