Research map / 3D vision / 3D generation
DenseMatcher: Learning 3D Semantic Correspondence for Category-Level Manipulation from a Single Demo
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DenseMatcher, which lifts multiview SD-DINO features onto meshes, refines them with a trainable DiffusionNet, and solves an improved functional map for 3D semantic correspondence, plus a new textured 3D matching dataset (DenseCorr3D) and one-demo robot manipulation.
Problem
Dense 3D semantic correspondence between textured, in-the-wild objects across categories, which prior shape-matching methods (geometry-only, category-specific, untextured) handle poorly; this limits one-demo robotic manipulation transfer.
Approach
Combine multiview 2D foundation-model features (SD-DINO, FeatUp) projected onto mesh vertices with a lightweight DiffusionNet refiner trained using a semantic-geodesic-distance loss, then compute correspondences via functional map with added entropy-sparsity and soft-assignment constraints.
Key results
- DenseMatcher reaches AUC 0.845 and Err 1.74 on DenseCorr3D full test set, versus 0.589 / 6.08 for the best baseline URSSM (DenseCorr3D).
- Real-world robot manipulation over six tasks achieves 76.7% overall success versus 50% for Robo-ABC with new memory.
- Ablations show removing the DiffusionNet refiner drops AUC from 0.845 to 0.672 on the full test set.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DenseMatcher instead of URSSM (DenseCorr3D) | 3D dense matching · DenseCorr3D All | AUC: 0.589 -> 0.845 | Helped |
| Use DenseMatcher instead of URSSM (DenseCorr3D) | 3D dense matching · DenseCorr3D Held-out | Err: 6.87 -> 2.82 | Helped |
| Add the DiffusionNet refiner (ablation) | DenseMatcher · DenseCorr3D All | AUC: 0.672 -> 0.845 | Helped |
| Add the feature preservation loss (ablation) | DenseMatcher · DenseCorr3D All | AUC: 0.568 -> 0.845 | Helped |
| Add FeatUp feature upsampling (ablation) | DenseMatcher · DenseCorr3D All | AUC: 0.741 -> 0.845 | Helped |
| Add entropy and sum-to-1 functional-map constraints (ablation) | DenseMatcher · DenseCorr3D All | AUC: 0.824 -> 0.845 | Helped |
| Use DenseMatcher instead of Robo-ABC with New Memory | real-world robot manipulation · Overall across six tasks | success rate: 50% -> 76.7% | Helped |
Surprising findings
- Removing the functional map regularization terms yields higher AUC drop-off only modestly (0.824 vs 0.845), but removing FeatUp gives lower Err than full model on the full set (3.48 vs 1.74 is worse) — mixed ablation effects are reported.
- Diff3F achieves lower Err (5.96) than URSSM on the full set despite lower AUC (0.522), indicating metric trade-offs.
Limitations
- Relies on textured meshes as input, so it is not applicable to untextured shapes without adaptation.
- Robot experiments are limited to six tasks with five trials each.
Experimental setup
Datasets: DenseCorr3D Objaverse-XL OmniObject3D FAUST ImageNet
Benchmarks: DenseCorr3D
Metrics: AUC Normalized geodesic error (Err) task success rate
Baselines: ConsistFMap URSSM Diff3F Robo-ABC
Builds on: functional map DiffusionNet SD-DINO FeatUp Diff3F
Closest papers by meaning
- Jamais Vu: Exposing the Generalization Gap in Supervised Semantic Correspondence (NeurIPS 2025 · similarity 0.87)
- Localizing, Structuring, and Rendering: Bridging 3D and 2D Vision-Language-Action Models for Robotic Manipulation (CVPR 2026 · similarity 0.87)
- UniGarmentManip: A Unified Framework for Category-Level Garment Manipulation via Dense Visual Correspondence (CVPR 2024 · similarity 0.86)
- G3Flow: Generative 3D Semantic Flow for Pose-aware and Generalizable Object Manipulation (CVPR 2025 · similarity 0.85)
- OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints (CVPR 2025 · similarity 0.85)
- Tango3D: Towards Alignment for Global and Local 2D-3D Correspondence (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.