Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Telling Left from Right: Identifying Geometry-Aware Semantic Correspondence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies geometry-aware semantic correspondence as a failure mode of foundation-model features (SD+DINO, DINOv2), and proposes test-time pose alignment plus a lightweight dense-trained post-processor with pose-variant augmentation and window soft-argmax, along with a new AP-10K-based benchmark.
Problem
Semantic correspondence features from pre-trained foundation models struggle to distinguish geometrically ambiguous matches (e.g., left vs. right paw) and are sensitive to pose variation.
Approach
Defines a geometry-aware subset of correspondences, shows foundation features are globally pose-aware, aligns viewpoints at test time via flip/rotation augmentations scored by instance matching distance, and trains a light post-processor with a soft-argmax dense loss, pose-variant (flip) augmentation, and window soft-argmax inference.
Key results
- Zero-shot method reaches 65.4 PCK@0.10 on SPair-71k, versus 59.9 for SD+DINO.
- Supervised method reaches 85.6 PCK@0.10 on SPair-71k (85.6 with AP-10K pre-training), up from 74.6 for SD+DINO (S).
- Geometry-aware subset gap reduced from 9.38% (SD+DINO (S)) to 3.86% on SPair-71k PCK@0.10.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add test-time pose alignment | SD+DINO zero-shot correspondence · SPair-71k | PCK@0.10: 59.9 -> 65.4 | Helped |
| add the proposed supervised post-processing method | SD+DINO · SPair-71k | PCK@0.10: 74.6 -> 82.9 | Helped |
| pre-train on AP-10K | Ours · SPair-71k | PCK@0.10: 82.9 -> 85.6 | Helped |
| add adaptive pose alignment | Ours · SPair-71k | PCK@0.10: 82.9 -> 83.2 | Helped |
| add adaptive pose alignment | Ours · PF-Pascal | PCK@0.10: 95.1 -> 95.0 | Hurt |
| add dense training objective (ablation) | SD+DINO (S) · SPair-71k standard subset | PCK@0.10: 74.6 -> 78.3 | Helped |
| add dense training objective (ablation) | SD+DINO (S) · SPair-71k geometry-aware subset | PCK@0.10: 67.6 -> 71.9 | Helped |
| add pose-variant augmentation (ablation) | SD+DINO (S) with dense training objective · SPair-71k standard subset | PCK@0.10: 78.3 -> 80.0 | Helped |
| add pose-variant augmentation (ablation) | SD+DINO (S) with dense training objective · SPair-71k geometry-aware subset | PCK@0.10: 71.9 -> 73.9 | Helped |
| add ground-truth perturbation and feature-map dropout (ablation) | SD+DINO (S) with dense training and pose-variant augmentation · SPair-71k standard subset | PCK@0.10: 80.0 -> 81.3 | Helped |
| add ground-truth perturbation and feature-map dropout (ablation) | SD+DINO (S) with dense training and pose-variant augmentation · SPair-71k geometry-aware subset | PCK@0.10: 73.9 -> 75.4 | Helped |
| replace argmax with soft-argmax inference (ablation) | SD+DINO (S) post-processor · SPair-71k standard subset | PCK@0.10: 81.3 -> 81.0 | Hurt |
Surprising findings
- Geometry-aware correspondences make up about 60% of SPair-71k matching keypoints, and state-of-the-art methods perform substantially worse on them.
- DINOv2 features are poor at left/right discrimination (63.8%) despite strong front/back accuracy, while SD+DINO nearly solves rough pose prediction.
Limitations
- Small instances are hard due to low resolution of raw feature maps.
- Fails under extreme pose variation combined with severe deformation.
Experimental setup
Datasets: SPair-71k PF-Pascal AP-10K
Benchmarks: SPair-71k PF-Pascal AP-10K
Metrics: PCK@0.01 PCK@0.05 PCK@0.10 PCK@0.15
Baselines: ASIC DINOv2+NN DIFT SD+DINO NeuCongeal SCOT PMNC SCorrSAN CATs++ DHF SD+DINO (S)
Builds on: SD+DINO (DIFT-style fused features) DINOv2 Stable Diffusion soft-argmax
Closest papers by meaning
- Jamais Vu: Exposing the Generalization Gap in Supervised Semantic Correspondence (NeurIPS 2025 · similarity 0.88)
- Improving Semantic Correspondence with Viewpoint-Guided Spherical Maps (CVPR 2024 · similarity 0.88)
- SemAlign3D: Semantic Correspondence between RGB-Images through Aligning 3D Object-Class Representations (CVPR 2025 · similarity 0.88)
- Bridging Viewpoint Gaps: Geometric Reasoning Boosts Semantic Correspondence (CVPR 2025 · similarity 0.88)
- SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation (NeurIPS 2025 · similarity 0.86)
- Multiview Equivariance Improves 3D Correspondence Understanding with Minimal Feature Finetuning (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.