Research map / 3D vision / 3D generation
Bridging Viewpoint Gaps: Geometric Reasoning Boosts Semantic Correspondence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes DUSt3R on synthetic cross-instance ShapeNet renderings to deform objects into a shared 3D shape, then matches via 3D KNN geometry followed by sparse DINOv2/SD semantic refinement for zero-shot semantic correspondence.
Problem
Semantic correspondence between images of the same category under large viewpoint changes, where 2D semantic features (SD, DINOv2) lack viewpoint invariance and dense feature matching is slow.
Approach
Establish coarse correspondences by geometric reasoning in an aligned 3D space: fine-tune DUSt3R to reconstruct two distinct instances and learn a deformation head that morphs one object's shape to the other's, supervised by DINOv2-derived sparse semantic pseudo-labels; then KNN geometric matching plus sparse semantic matching within local candidates.
Key results
- Geo-only variant reaches 69.6 PCK@0.1 on rigid SPair-71k, +4.7 over Telling Left from Right (64.9).
- Geo + DINOv2 + SD semantic refinement reaches 72.3 PCK@0.1 (+7.4 over best prior), and 19.1 PCK@0.01 vs 9.0 for prior best.
- On the large-viewpoint subset, geo-only gets 68.8 vs 52.3 for Telling Left from Right (+16.5); with semantic refinement 71.9; geometry-only runs about 40x faster (19.9 fps).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DINOv2 ViT-S semantic matching to geometric matching (ablation) | Ours (Geo only) · SPair-71k rigid-body test | PCK@0.1: 69.6 -> 71.5 | Helped |
| Add DINOv2 ViT-S semantic matching to geometric matching (ablation) | Ours (Geo only) · SPair-71k rigid-body test | PCK@0.05: 53.1 -> 56.4 | Helped |
| Add DINOv2 ViT-S semantic matching to geometric matching (ablation) | Ours (Geo only) · SPair-71k rigid-body test | PCK@0.01: 11.4 -> 15.1 | Helped |
| Replace DINOv2 ViT-S features with fused DINOv2 ViT-B and Stable Diffusion features (ablation) | Ours (Geo + Semantic (DINOv2 ViT-S)) · SPair-71k rigid-body test | PCK@0.1: 71.5 -> 72.3 | Helped |
| Replace DINOv2 ViT-S features with fused DINOv2 ViT-B and Stable Diffusion features (ablation) | Ours (Geo + Semantic (DINOv2 ViT-S)) · SPair-71k rigid-body test | PCK@0.05: 56.4 -> 59.1 | Helped |
| Replace DINOv2 ViT-S features with fused DINOv2 ViT-B and Stable Diffusion features (ablation) | Ours (Geo + Semantic (DINOv2 ViT-S)) · SPair-71k rigid-body test | PCK@0.01: 15.1 -> 19.1 | Helped |
| Use DINOv2 ViT-B and Stable Diffusion semantic features instead of DINOv2 ViT-S (ablation) | Ours (Geo + Semantic) · SPair-71k rigid-body test | fps: 5.4 -> 0.4 | Hurt |
Surprising findings
- Geometry-only matching without any semantic features outperforms prior methods that use semantic features with geometric enhancement, especially under extreme viewpoint change.
Limitations
- Restricted to rigid-body categories; non-rigid animals lack consistent 3D data and canonical orientation.
- Performance depends on ShapeNet mesh quality; bus category suffers from simplistic meshes and front/back ambiguity.
- Sim-to-real gap from synthetic training data.
Experimental setup
Datasets: ShapeNetCore SPair-71k
Benchmarks: SPair-71k
Metrics: PCK@0.10 PCK@0.05 PCK@0.01 fps latency
Baselines: DINOv2+NN DIFT SD+DINO Spherical Maps Spherical Maps + SD Telling Left from Right
Builds on: DUSt3R DINOv2 Telling Left from Right Spherical Maps DIFT
Closest papers by meaning
- Improving Semantic Correspondence with Viewpoint-Guided Spherical Maps (CVPR 2024 · similarity 0.89)
- Jamais Vu: Exposing the Generalization Gap in Supervised Semantic Correspondence (NeurIPS 2025 · similarity 0.88)
- Telling Left from Right: Identifying Geometry-Aware Semantic Correspondence (CVPR 2024 · similarity 0.88)
- SemAlign3D: Semantic Correspondence between RGB-Images through Aligning 3D Object-Class Representations (CVPR 2025 · similarity 0.87)
- Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning (CVPR 2024 · similarity 0.87)
- G^3-LQ: Marrying Hyperbolic Alignment with Explicit Semantic-Geometric Modeling for 3D Visual Grounding (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.