Research map / Point clouds / Point cloud classification
Lift3D Policy: Lifting 2D Foundation Models for Robust 3D Robotic Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Lift3D enhances 2D foundation models (CLIP, DINOv2) for 3D robotic manipulation via a task-aware masked autoencoder that reconstructs depth on affordance patches, then lifts the model to encode point clouds by reusing its pretrained 2D position embeddings.
Problem
Robotic manipulation policies need 3D geometric understanding, but explicit 3D encoders lack large-scale pretraining data and modality transformations (projection/lifting) lose spatial information.
Approach
Two-stage enhancement of a transformer 2D foundation model: (1) task-aware MAE masking CLIP-attention-guided affordance patches and reconstructing depth with distillation; (2) map 3D point-cloud tokens onto virtual cube-face planes and average the corresponding pretrained 2D position embeddings to feed point clouds directly into the frozen 2D model for imitation learning.
Key results
- Lift3D (CLIP) reaches 83.9% mean success on 15 MetaWorld tasks vs 65.3% for DP3 and 75.1% for R3M baselines.
- Lift3D achieves 62.5% mean real-world success over 10 tasks vs 45.5% for DP3.
- Scaling DINOv2 backbone to ViT-giant raises shelf-place success from 28% to 68%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Lift3D (DINOV2) instead of R3M | MetaWorld policy · MetaWorld | Mean success rate: 75.1 -> 84.5 | Helped |
| Use Lift3D (CLIP) instead of R3M | Adroit policy · Adroit | Mean success rate: 85.3 -> 88.0 | Helped |
| Use Lift3D instead of DP3 | Real-world manipulation policy · Real-world tasks | Mean success rate: 45.5 -> 62.5 | Helped |
| Add depth reconstruction to the baseline configuration (ablation) | Lift3D · MetaWorld | Mean success rate: 65.3 -> 68.1 | Helped |
| Reconstruct RGB instead of depth (ablation) | Task-aware MAE · MetaWorld | Mean success rate: 65.3 -> 65.4 | No clear effect |
| Add affordance-guided masking to depth reconstruction (ablation) | Task-aware MAE · MetaWorld | Mean success rate: 68.1 -> 71.9 | Helped |
| Add visual token distillation (ablation) | Task-aware MAE with affordance-guided depth reconstruction · MetaWorld | Mean success rate: 71.9 -> 76.5 | Helped |
| Add 2D model-lifting with pretrained 2D positional embeddings (ablation) | Lift3D · MetaWorld | Mean success rate: 65.3 -> 74.9 | Helped |
| Add implicit 3D representation learning to 2D model-lifting (ablation) | Lift3D · MetaWorld | Mean success rate: 74.9 -> 83.9 | Helped |
| Replace pretrained 2D positional embeddings with newly introduced learnable embeddings (ablation) | 2D model-lifting · MetaWorld | Mean success rate: 83.9 -> 79.2 | Hurt |
| Scale the foundation model from DINOV2-ViT-base to DINOV2-ViT-large | Lift3D · Shelf-place | Success rate: 28 -> 58 | Helped |
| Scale the foundation model from DINOV2-ViT-base to DINOV2-ViT-giant | Lift3D · Shelf-place | Success rate: 28 -> 68 | Helped |
Surprising findings
- Random RGB reconstruction gives little gain over the baseline, while depth reconstruction yields a clear improvement.
- Training the lifted 2D PEs from scratch instead of using pretrained 2D PEs drops performance by 6 points on MetaWorld.
Limitations
- Cannot comprehend language conditions; policy is not language-conditioned.
- Real-world evaluation limited to a single camera setup and 10 tasks with 20 trials each.
Experimental setup
Datasets: large-scale robotic manipulation video datasets with paired depth (1M samples) MetaWorld demonstrations Adroit demonstrations RLBench demonstrations real-world Franka demonstrations
Benchmarks: MetaWorld Adroit RLBench real-world Franka tasks
Metrics: success rate
Baselines: CLIP R3M VC-1 PointNet PointNet++ PointNeXt SPA DP3
Builds on: MAE DINOv2 CLIP DP3 RVT-2
Closest papers by meaning
- 3D-LFM: Lifting Foundation Model (CVPR 2024 · similarity 0.87)
- DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation (NeurIPS 2025 · similarity 0.87)
- Lift3D: Zero-Shot Lifting of Any 2D Vision Model to 3D (CVPR 2024 · similarity 0.87)
- GravMAD: Grounded Spatial Value Maps Guided Action Diffusion for Generalized 3D Manipulation (ICLR 2025 · similarity 0.86)
- 3D-MVP: 3D Multiview Pretraining for Manipulation (CVPR 2025 · similarity 0.86)
- SUGAR: Pre-training 3D Visual Representations for Robotics (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.