Research map / 3D vision / 3D generation
From Thousands to Billions: 3D Visual Language Grounding via Render-Supervised Distillation from 2D VLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LIFT-GS, which pretrains a 3D vision-language grounding decoder by rendering predicted 3D Gaussian masks into 2D views and supervising them with SAM/CLIP pseudo-labels, then fine-tunes on 3D annotations.
Problem
3D vision-language grounding is limited by scarce labeled 3D scenes (thousands) compared with billions of 2D images, so 3D grounding models underperform and are data-starved.
Approach
Use differentiable rendering of feed-forward predicted 3D Gaussians and language-conditioned 3D masks so that 2D losses from SAM-CLIP pseudo-labels supervise the entire encoder-decoder end-to-end, without any 3D annotations during pretraining.
Key results
- Open-vocabulary instance segmentation on ScanNet200: LIFT-GS reaches 25.7 mAP vs 20.2 for PQ3D (prior SOTA).
- 3D referential grounding on sensor point clouds: LIFT-GS reaches 50.9 Acc@25 on SR3D, 43.7 on NR3D and 49.7 on ScanRefer, with relative gains of 10-30% over the scratch-trained baseline and BUTD-DETR.
- Fine-tuning scaling: a pretrained model with 50% of fine-tuning data matches training from scratch with 100% data (roughly 2x data multiplier).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pretrain with LIFT-GS before 3D fine-tuning | LIFT-GS · ScanNet200 | mAP: 22.5 -> 25.7 | Helped |
| pretrain with LIFT-GS before 3D fine-tuning | LIFT-GS · SR3D (Sensor PC) | Acc@25: 44.0 -> 50.9 | Helped |
| pretrain with LIFT-GS before 3D fine-tuning | LIFT-GS · NR3D (Sensor PC) | Acc@25: 37.2 -> 43.7 | Helped |
| pretrain with LIFT-GS before 3D fine-tuning | LIFT-GS · ScanRefer (Sensor PC) | Acc@50: 29.5 -> 36.4 | Helped |
| use SAM-CLIP pseudo-labels instead of official Ponder-v2 pretraining | Ponder-v2 · Combined ScanRefer, SR3D, NR3D | Acc@0.25: 40.92 -> 45.40 | Helped |
| use LIFT-GS multimodal decoder rather than Ponder-v2† | 3D referential grounding · Combined ScanRefer, SR3D, NR3D | Acc@0.25: 45.40 -> 47.53 | Helped |
| add grounding loss to RGB and feature pretraining losses (ablation) | LIFT-GS pretraining · Combined ScanRefer, SR3D, NR3D | Acc@0.5: 31.35 -> 33.75 | Helped |
| add RGB reconstruction loss to grounding-loss pretraining (ablation) | LIFT-GS pretraining · Combined ScanRefer, SR3D, NR3D | Acc@0.25: 46.34 -> 46.67 | Helped |
| add feature loss to grounding and RGB pretraining losses (ablation) | LIFT-GS pretraining · Combined ScanRefer, SR3D, NR3D | Acc@0.75: 12.45 -> 13.49 | Helped |
| add ScanNet++ to pretraining data (ablation) | LIFT-GS · Combined ScanRefer, SR3D, NR3D | Acc@0.25: 47.53 -> 48.29 | Helped |
| add Taskonomy and Aria Synthetic to ScanNet and ScanNet++ pretraining (ablation) | LIFT-GS · Combined ScanRefer, SR3D, NR3D | Acc@0.25: 48.29 -> 48.49 | Helped |
| replace SAM-H and CLIP-L pseudo-label models with SAM-B and CLIP-B (ablation) | LIFT-GS pretraining · Combined ScanRefer, SR3D, NR3D | Acc@0.5: 33.75 -> 31.50 | Hurt |
Surprising findings
- Grounding loss alone (no photometric or feature loss) yields most of the pretraining gain, showing the Gaussian decoder can be trained as a latent without direct supervision.
- Captions from LLaMA-3.2 Vision on SAM segments as pseudo-queries perform comparably to CLIP embeddings for pretraining.
Limitations
- Zero-shot 3D segmentation from 2D pseudo-labels alone is low accuracy; CLIP image embeddings used as text queries are imperfect.
- Still behind two-stage 3D-VisTA at Acc@0.5 on referential grounding.
- Relies on pseudo-label quality from 2D foundation models, which hallucinate multi-view spatial relations.
Experimental setup
Datasets: ScanNet ScanNet200 ScanRefer SR3D NR3D ScanNet++ Taskonomy Aria Synthetic
Benchmarks: ScanNet200 ScanRefer SR3D NR3D
Metrics: mAP mAP25 mAP50 Acc@0.25 Acc@0.5 Acc@0.75
Baselines: OpenScene OpenMask3D PQ3D BUTD-DETR 3D-VisTA PonderV2 LanguageRefer SAT-2D
Builds on: UniVLG PonderV2 MaskFormer MDETR SAM
Closest papers by meaning
- Unifying 2D and 3D Vision-Language Understanding (ICML 2025 · similarity 0.91)
- Towards CLIP-driven Language-free 3D Visual Grounding via 2D-3D Relational Enhancement and Consistency (CVPR 2024 · similarity 0.90)
- Naturally Supervised 3D Visual Grounding with Language-Regularized Concept Learners (CVPR 2024 · similarity 0.90)
- MonoVLM: Monocular 3D Visual Grounding with Vision Language Models (CVPR 2026 · similarity 0.89)
- G^3-LQ: Marrying Hyperbolic Alignment with Explicit Semantic-Geometric Modeling for 3D Visual Grounding (CVPR 2024 · similarity 0.89)
- Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.