Research map / 3D vision / 3D generation
Jamais Vu: Exposing the Generalization Gap in Supervised Semantic Correspondence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a canonical 3D prototype method that lifts 2D keypoints to 3D via monocular depth, adds geometric alignment loss on top of Geo-SC, and introduces SPair-U, a benchmark with new unseen keypoint annotations to test generalization in supervised semantic correspondence.
Problem
Supervised semantic correspondence methods overfit to sparsely annotated training keypoints and generalize poorly to unseen keypoints, behaving like keypoint detectors.
Approach
Lift annotated 2D keypoints to a shared 3D canonical prototype using monocular depth (MoGe) and rigid Procrustes alignment, then train dense features with a local geometric alignment loss between pseudo-geodesic neighborhoods in posed and canonical space, yielding a continuous canonical manifold without explicit 3D or camera supervision.
Key results
- On SPair-U (unseen keypoints), the method averages 62.4 PCK@0.1 vs 56.9 for Geo-SC, and 66.1 with AP-10K pretraining, a 6.1% gain over the best prior supervised model.
- On standard SPair-71k test, the method averages 82.9 vs 83.2 for Geo-SC (marginal 0.2% drop).
- Cross-benchmark: Geo-SC underperforms DINO+SD (S) at PCK@0.10 on SPair-U (56.9 vs 59.3 for the baseline per the cross-benchmark table? baseline 59.3 vs Geo-SC 56.9) while the proposed model reaches 62.4 on SPair-U.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the canonical prototype to sparse DINOv2+SD supervision (ablation) | DINOv2+SD supervised baseline · SPair-U | PCK@0.1: 60.0 -> 66.0 | Helped |
| Add the canonical prototype to sparse DINOv2+SD supervision (ablation) | DINOv2+SD supervised baseline · SPair-71k test | PCK@0.1: 76.5 -> 75.9 | Hurt |
| Add Geo-SC-specific losses to sparse DINOv2+SD supervision (ablation) | DINOv2+SD supervised baseline · SPair-U | PCK@0.1: 60.0 -> 57.1 | Hurt |
| Combine Geo-SC losses with the canonical prototype (ablation) | DINOv2+SD supervised baseline · SPair-U | PCK@0.1: 60.0 -> 66.1 | Helped |
| Use AP-10K pretraining | Ours · SPair-U unseen keypoints | PCK@0.1 bbox: 62.4 -> 66.1 | Helped |
| Set λZ to 1 instead of the full-model setting (ablation) | Full model · SPair-71k validation | PCK@0.1: 86.5 -> 85.9 | Hurt |
| Set λZ to 0.1 instead of the full-model setting (ablation) | Full model · SPair-71k validation | PCK@0.1: 86.5 -> 86.1 | Hurt |
| Vary the geometric-loss neighborhood size (ablation) | Full model · SPair-71k validation | PCK@0.1: 85.5; 86.1; 85.9; 86.5; 86.5; 86.6; 86.1 | No clear effect |
| Use a local subset of keypoints rather than globally aligning all keypoints (ablation) | Canonical prototype learning · SPair-71k validation | PCK | No clear effect |
| Add the proposed geometric canonical-manifold method to Geo-SC | Geo-SC · SPair-71k test | PCK@0.1 bbox: 83.2 -> 82.9 | No clear effect |
| Add the proposed geometric canonical-manifold method to Geo-SC | Geo-SC · SPair-U unseen keypoints | PCK@0.1 bbox: 56.9 -> 62.4 | Helped |
| Add the proposed geometric canonical-manifold method to Geo-SC | Geo-SC · AP-10K CS | PCK@0.10: 67.7 -> 69.0 | Helped |
Surprising findings
- Supervised SC models trained on SPair-71k often underperform unsupervised DINOv2+SD on unseen keypoints, and Geo-SC loses to its unsupervised counterpart on SPair-U.
- In eight categories, the best-performing model had not seen the evaluated keypoints during training, suggesting supervised methods act as keypoint regressors.
Limitations
- SPair-U test set is small (481 images, about 8,000 pairs) and restricted to common objects and animals.
- Global rigid alignment assumption is coarse for deformable objects, and geometry may not proxy semantics for categories with diverse part configurations such as cabinets.
- Requires additional depth maps and segmentation masks during training, unlike purely keypoint-supervised methods.
Experimental setup
Datasets: SPair-71k AP-10K PF-PASCAL
Benchmarks: SPair-71k SPair-U AP-10K PF-PASCAL
Metrics: PCK@0.1 bbox PCK@0.01 PCK@0.05 PCK^dagger KAP geo-aware PCK
Baselines: SD DINOv2 DINOv2+SD SphericalMaps SCorrSan CATS++ DHF DINO+SD (S) Geo-SC
Builds on: Geo-SC SphericalMaps DINOv2+SD Kabsch-Umeyama
Closest papers by meaning
- Improving Semantic Correspondence with Viewpoint-Guided Spherical Maps (CVPR 2024 · similarity 0.90)
- SemAlign3D: Semantic Correspondence between RGB-Images through Aligning 3D Object-Class Representations (CVPR 2025 · similarity 0.89)
- Telling Left from Right: Identifying Geometry-Aware Semantic Correspondence (CVPR 2024 · similarity 0.88)
- Bridging Viewpoint Gaps: Geometric Reasoning Boosts Semantic Correspondence (CVPR 2025 · similarity 0.88)
- DenseMatcher: Learning 3D Semantic Correspondence for Category-Level Manipulation from a Single Demo (ICLR 2025 · similarity 0.87)
- One-shot 3D Object Canonicalization based on Geometric and Semantic Consistency (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.