Research map / 3D vision / 3D generation
C3Po: Cross-View Cross-Modality Correspondence by Pointmap Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the C3 dataset of floor plan-photo pairs with pixel correspondences and camera poses built from SfM reconstructions aligned to floor plans, and C3Po, a DUSt3R fine-tune that predicts cross-view cross-modal correspondences.
Problem
Predicting pixel correspondences between ground-level photos and abstract floor plans is hard for existing matchers because viewpoint and modality both change, and no prior dataset provided ground-truth correspondences for this setting.
Approach
Build correspondences by aligning COLMAP point clouds from Internet photo collections to Wikimedia floor plans, then fine-tune DUSt3R with split encoders for each modality and pointmap prediction projected orthographically onto the floor plan.
Key results
- C3Po reduces RMSE by 34% relative to the best-performing baseline on the C3 test set.
- Camera pose recall for C3Po is 21.86% at 5 degrees and 51.35% at 30 degrees angular threshold.
- Baseline correspondence methods (DINOv2, DIFT, RoMa, LoFTR, SuperGlue, MASt3R, DUSt3R) yield errors often above 10% of image size.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune DUSt3R on C3 using pointmap prediction for floor plan-photo correspondence | DUSt3R · C3 test set | RMSE: 34% decrease | Helped |
| Discard the z-coordinate instead of the y-coordinate in orthographic projection (ablation) | C3Po · C3 correspondence training | model convergence speed | Hurt |
Surprising findings
- MASt3R, built on DUSt3R for matching, shows higher error than DUSt3R on this task, which the authors attribute to DUSt3R predicting scene structure that is projected onto the floor plan.
Limitations
- Ambiguity from minimal-context photos leads to unimodal wrong predictions.
- Structural symmetry causes multiple plausible correspondence alignments.
- Errors remain much higher than classical correspondence problems.
Experimental setup
Datasets: C3 MegaScenes YFCC100M Wikimedia Commons floor plans
Benchmarks: C3
Metrics: RMSE PCK precision-recall recall at angular and positional thresholds
Baselines: SuperGlue LoFTR DINOv2 DIFT RoMa MASt3R DUSt3R
Builds on: DUSt3R COLMAP MASt3R
Closest papers by meaning
- CrossOver: 3D Scene Cross-Modal Alignment (CVPR 2025 · similarity 0.87)
- Bridging Viewpoint Gaps: Geometric Reasoning Boosts Semantic Correspondence (CVPR 2025 · similarity 0.86)
- Self-Supervised Spatial Correspondence Across Modalities (CVPR 2025 · similarity 0.86)
- DUSt3R: Geometric 3D Vision Made Easy (CVPR 2024 · similarity 0.85)
- MvFFN: Multi-view Floor-Plan Feed-Forward Network for Unposed Wide-Baseline Panorama Layout Reconstruction (NeurIPS 2026 · similarity 0.85)
- Unposed Sparse Views Room Layout Reconstruction in the Age of Pretrain Model (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.