Research map / 3D vision / 3D generation
Multiview Equivariance Improves 3D Correspondence Understanding with Minimal Feature Finetuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Evaluates 3D view-equivariance of ViT foundation models (DINOv2 and others) on Objaverse and MVImgNet correspondences, then finetunes them with LoRA plus one conv layer using a SmoothAP loss on multiview correspondences, improving pose estimation, tracking, and semantic correspondence.
Problem
Vision foundation models are trained mainly on 2D images, and it is unclear how well their features capture 3D structure, i.e., whether features are consistent across views of the same 3D point and how this affects 3D correspondence tasks.
Approach
Measure multiview feature equivariance with dense correspondences rendered from Objaverse and MVImgNet, show it correlates with downstream 3D correspondence performance, and finetune ViTs (LoRA in last four blocks plus a single 3x3 conv layer) with a SmoothAP ranking loss on corresponding pixels from two random views; even one object for one iteration yields gains.
Key results
- DINOv2 finetuning improves 3cm-3deg pose accuracy by 9.58 on OnePose-LowTex, 5.0 Average Jaccard on TAP-VID-DAVIS, and 5.06 PCK@0.05 on PF-PASCAL (different viewpoints).
- Finetuning on a single Objaverse object for one iteration improves DINOv2 by 4.85 (3cm-3deg), 3.55 (AJ), and 3.47 (PCK@0.05).
- On Objaverse and MVImgNet, DINOv2 is the strongest of the evaluated ViTs in 3D equivariance; finetuning improves PCDP@0.05 on MVImgNet from 62.09 to 71.74.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| finetune DINOv2 with multiview correspondence SmoothAP | DINOv2 base · OnePose-LowTex | 3cm-3deg: 48.45 -> 58.03 | Helped |
| finetune DINOv2 with multiview correspondence SmoothAP | DINOv2 base · TAP-Vid-DAVIS | AJ: 40.40 -> 46.85 | Helped |
| finetune DINOv2 with multiview correspondence SmoothAP | DINOv2 base · PF-PASCAL (Different Viewpoints) | PCK0.05: 42.18 -> 47.24 | Helped |
| add one convolution layer to the finetuning architecture (ablation) | DINOv2 feature finetuning · OnePose-LowTex | 3cm-3deg: 53.85 -> 58.03 | Helped |
| use two convolution layers instead of one (ablation) | DINOv2 feature finetuning · OnePose-LowTex | 3cm-3deg: 58.03 -> 56.14 | Hurt |
| finetune on MVImgNet instead of Objaverse (ablation) | DINOv2 · PF-PASCAL (Different Viewpoints) | PCK0.05: 47.25 -> 45.13 | Hurt |
| finetune on scene-centric data instead of Objaverse (ablation) | DINOv2 · PF-PASCAL (Different Viewpoints) | PCK0.05: 47.25 -> 41.73 | Hurt |
| replace SmoothAP with contrastive loss (ablation) | DINOv2 feature finetuning · OnePose-LowTex | 3cm-3deg: 58.03 -> 55.57 | Hurt |
| replace SmoothAP with differentiable Procrustes loss (ablation) | DINOv2 feature finetuning · OnePose-LowTex | 3cm-3deg: 58.03 -> 55.00 | Hurt |
| concatenate features from layers 2, 5, 8, and 11 instead of using layer 11 (ablation) | DINOv2 feature finetuning · TAP-Vid-DAVIS | AJ: 46.85 -> 39.67 | Hurt |
| use learning rate 1e-6 instead of 1e-5 (ablation) | DINOv2 feature finetuning · OnePose-LowTex | 3cm-3deg: 58.03 -> 55.03 | Hurt |
| replace DINOv2 features in Wild Gaussians with finetuned DINO features | Wild Gaussians · Mountain | PSNR: 20.82 -> 21.01 | Helped |
Surprising findings
- Finetuning on a single untextured hemisphere for one iteration yields substantial gains in 3D correspondence.
- Finetuning on scene-centric datasets (RealEstate10K, Spaces, LLFF) does not help and sometimes hurts performance on PF-PASCAL.
Limitations
- Semantic-level tasks such as segmentation, depth, and instance recognition show performance on par or slightly worse than the baseline.
- ConvNeXt features remain weaker than modern ViT features even after finetuning.
Experimental setup
Datasets: Objaverse MVImgNet
Benchmarks: OnePose-LowTexture YCB-Video TAP-Vid-DAVIS PF-PASCAL Objaverse correspondence MVImgNet correspondence Paris-H VOC2012 NYUv2
Metrics: APE PCDP 3cm-3deg pose accuracy Average Jaccard PCK@0.05 AR
Baselines: FiT FiT-Reg DUSt3R OnePose++ MegaPose Co-Tracker
Builds on: SmoothAP LoRA DINO-Tracker FiT
Closest papers by meaning
- GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models (ICLR 2026 · similarity 0.87)
- ImageNet3D: Towards General-Purpose Object-Level 3D Understanding (NeurIPS 2024 · similarity 0.87)
- Probing the 3D Awareness of Visual Foundation Models (CVPR 2024 · similarity 0.87)
- Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering (NeurIPS 2026 · similarity 0.86)
- Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning (CVPR 2026 · similarity 0.86)
- Improving Semantic Correspondence with Viewpoint-Guided Spherical Maps (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.