Research map / 3D vision / 3D generation
SemAlign3D: Semantic Correspondence between RGB-Images through Aligning 3D Object-Class Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds category-level 3D object-class representations from monocular depth (DepthAnythingV2) and GeoAware features, then aligns them to RGB instances via gradient-descent alignment energy for sparse semantic correspondence.
Problem
Semantic correspondence between images with extreme view variation, where large vision model features capture local semantics but not global geometric relations between object parts.
Approach
Learn sparse-then-dense 3D object-class point clouds from a few sparsely annotated images using monocular depth and semantic features, then align the 3D model to the target image by minimizing a likelihood-based alignment loss with geometric consistency terms, and transfer correspondences through the aligned model.
Key results
- Overall PCK@0.1 on SPair-71k rises from 85.6% (GeoAware) to 88.9%, with gains above 10 points on Bottle, Chair, and TV.
- PCK@0.05 on SPair-71k improves from 75.3% to 77.5%, while PCK@0.01 drops from 22.0% to 15.8%.
- On AP-10K subset and PF-PASCAL the method underperforms GeoAware (e.g., AP-10K PCK@0.1 89.1 vs 92.8), attributed to spatial bias and non-rigid objects.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 3D object-class representations and alignment instead of GeoAware alone | GeoAware semantic correspondence · SPair-71k | PCK@0.1: 85.6 -> 88.9 | Helped |
| Use 3D object-class representations and alignment instead of GeoAware alone | GeoAware semantic correspondence · SPair-71k | PCK@0.01: 22.0 -> 15.8 | Hurt |
| Use 3D object-class representations and alignment instead of GeoAware alone | GeoAware semantic correspondence · SPair-71k | PCK@0.05: 75.3 -> 77.5 | Helped |
| Use 3D object-class representations and alignment instead of GeoAware alone | GeoAware semantic correspondence · Bottle, Chair, TV | PCK@0.1: 70.5 -> 82.2; 73.4 -> 88.3; 85.3 -> 96.1 | Mixed |
| Replace fine-tuned GeoAware backbone features with DINOv2+SD backbone features (ablation) | SemAlign3D · Bottle, Chair, TV | PCK@0.1: 82.2 -> 60.2; 88.3 -> 68.7; 96.1 -> 64.3 | Hurt |
| Add sparse reconstruction loss (ablation) | SemAlign3D · Bottle, Chair, TV | PCK@0.1: 79.1 -> 82.2; 84.2 -> 88.3; 92.1 -> 96.1 | Helped |
| Add dense reconstruction loss (ablation) | SemAlign3D · Bottle, Chair, TV | PCK@0.1: 64.0 -> 82.2; 63.0 -> 88.3; 85.9 -> 96.1 | Helped |
| Add background mask penalization loss (ablation) | SemAlign3D · Bottle, Chair, TV | PCK@0.1: 80.5 -> 82.2; 83.5 -> 88.3; 94.8 -> 96.1 | Mixed |
| Add depth regularization loss (ablation) | SemAlign3D · Bottle, Chair, TV | PCK@0.1: 81.4 -> 82.2; 85.1 -> 88.3; 94.3 -> 96.1 | Mixed |
Surprising findings
- Replacing the fine-tuned GeoAware features with unrefined DINOv2+SD features drops Bottle, Chair, and TV accuracy substantially, yet the method still beats other unsupervised baselines.
- The method is weaker on animal categories and on AP-10K and PF-PASCAL, and underperforms on PCK@0.01 and on PCK†/KAP at some thresholds.
Limitations
- Alignment is slow (10-30 s per image) due to per-image optimization.
- Pipeline is not end-to-end differentiable; representation learning and alignment are separate.
- Weaker on non-rigid animals and symmetric objects; 3D representations poorly capture joint limits and symmetry.
Experimental setup
Datasets: SPair-71k AP-10K PF-PASCAL
Benchmarks: SPair-71k AP-10K PF-PASCAL
Metrics: PCK@0.1 PCK@0.05 PCK@0.01 PCK† KAP
Baselines: GeoAware SphericalMaps DHF CATs++ SCorrSAN PMNC SCOT ASIC DIFT SD+DINOv2
Builds on: GeoAware DepthAnythingV2 DINOv2 Stable Diffusion features
Closest papers by meaning
- Jamais Vu: Exposing the Generalization Gap in Supervised Semantic Correspondence (NeurIPS 2025 · similarity 0.89)
- Improving Semantic Correspondence with Viewpoint-Guided Spherical Maps (CVPR 2024 · similarity 0.89)
- Pixel-level Semantic Correspondence through Layout-aware Representation Learning and Multi-scale Matching Integration (CVPR 2024 · similarity 0.89)
- Telling Left from Right: Identifying Geometry-Aware Semantic Correspondence (CVPR 2024 · similarity 0.88)
- Tango3D: Towards Alignment for Global and Local 2D-3D Correspondence (NeurIPS 2026 · similarity 0.87)
- Bridging Viewpoint Gaps: Geometric Reasoning Boosts Semantic Correspondence (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.