Research map / 3D vision / 3D generation
Lift3D: Zero-Shot Lifting of Any 2D Vision Model to 3D
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Lift3D trains a generalizable view-synthesis network on DINO and CLIP feature maps so that any 2D vision model's intermediate features can be rendered view-consistently in 3D zero-shot, then decoded by the model's own decoder.
Problem
2D vision model outputs are inconsistent across views, and lifting each 2D operator to 3D typically needs task-specific design and per-scene optimization.
Approach
Render intermediate encoder features of any 2D model via epipolar aggregation of RGB-guided blending weights, with a corrective feature aggregation module that uses RGB multi-view consistency to fix feature inconsistencies; the model is trained on only DINO and CLIP and generalizes zero-shot to other operators.
Key results
- Semantic segmentation: Lift3D IoU 0.871 on shoe rack vs ISRF 0.861 and N3F 0.589, comparable to scene-specific methods.
- 3D style transfer: Lift3D achieves best short-range RMSE 0.053 and long-range RMSE 0.138 among compared methods, with competitive LPIPS.
- Zero-shot lifting to open-vocabulary segmentation (OVSeg) and colorization (DDColor) without task-specific training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Lift3D instead of ISRF for stroke-guided segmentation | DINO ViT-S/8 features · Color Fountain | IoU: 0.927 -> 0.935 | Helped |
| Use Lift3D instead of ISRF for stroke-guided segmentation | DINO ViT-S/8 features · Color Fountain | mAP: 0.927 -> 0.938 | Helped |
| Use Lift3D instead of ISRF for stroke-guided segmentation | DINO ViT-S/8 features · Shoe Rack | IoU: 0.861 -> 0.871 | Helped |
| Use Lift3D instead of ISRF for stroke-guided segmentation | DINO ViT-S/8 features · Shoe Rack | mAP: 0.869 -> 0.875 | Helped |
| Use Lift3D instead of ISRF for stroke-guided segmentation | DINO ViT-S/8 features · Chess Table | IoU: 0.912 -> 0.824 | Hurt |
| Use Lift3D instead of ISRF for stroke-guided segmentation | DINO ViT-S/8 features · Chess Table | mAP: 0.916 -> 0.83 | Hurt |
| Use Lift3D instead of ISRF for stroke-guided segmentation | DINO ViT-S/8 features · Stove | IoU: 0.819 -> 0.817 | Hurt |
| Use Lift3D instead of ISRF for stroke-guided segmentation | DINO ViT-S/8 features · Stove | mAP: 0.817 -> 0.814 | Hurt |
| Use Lift3D instead of Hyper for 3D style transfer | CAST style-transfer features · LLFF scenes, short-range consistency | LPIPS: 0.158 -> 0.181 | Hurt |
| Use Lift3D instead of Hyper for 3D style transfer | CAST style-transfer features · LLFF scenes, short-range consistency | RMSE: 0.066 -> 0.053 | Helped |
| Use Lift3D instead of Hyper for 3D style transfer | CAST style-transfer features · LLFF scenes, long-range consistency | LPIPS: 0.310 -> 0.327 | Hurt |
| Use Lift3D instead of Hyper for 3D style transfer | CAST style-transfer features · LLFF scenes, long-range consistency | RMSE: 0.145 -> 0.138 | Helped |
Surprising findings
- Lifted OVSeg features even improve segmentation quality over 2D OVSeg (precise keyboard boundaries).
- Lift3D lifts a DINO ViT-S/8 variant unseen during training and still performs comparably to methods using larger DINO features.
Limitations
- View interpolation can lose visual quality compared with direct 2D operators.
- Style transfer and colorization evaluations are largely qualitative; metrics do not fully capture stylization quality.
- Semantic segmentation evaluated only on four scenes with user strokes.
Experimental setup
Datasets: IBRNet training data LLFF
Benchmarks: LLFF ISRF scenes (chess table, color fountain, stove, shoe rack)
Metrics: IoU mAP LPIPS RMSE
Baselines: N3F ISRF AdaIN CCPL ReReVST LSNV Hyper StyleRF CLIP-NeRF NeRF-Art Masked SDS InstructNeRF2NeRF
Builds on: IBRNet NeRF DFF InstructNeRF2NeRF
Closest papers by meaning
- Rethinking End-to-End 2D to 3D Scene Segmentation in Gaussian Splatting (CVPR 2025 · similarity 0.87)
- Lift3D Policy: Lifting 2D Foundation Models for Robust 3D Robotic Manipulation (CVPR 2025 · similarity 0.87)
- SAM2Object: Consolidating View Consistency via SAM2 for Zero-Shot 3D Instance Segmentation (CVPR 2025 · similarity 0.87)
- Unifying 2D and 3D Vision-Language Understanding (ICML 2025 · similarity 0.87)
- From Thousands to Billions: 3D Visual Language Grounding via Render-Supervised Distillation from 2D VLMs (ICML 2025 · similarity 0.87)
- ImageNet3D: Towards General-Purpose Object-Level 3D Understanding (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.