Research map / 3D vision / 3D generation
DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DistillNeRF, a generalizable feed-forward driving-scene model that distills per-scene EmerNeRF depth/virtual-view targets and CLIP/DINOv2 features into a sparse hierarchical voxel representation for single-timestep multi-camera input.
Problem
Learning 3D scene representations for outdoor autonomous driving from single-timestep, sparse, limited-overlap multi-camera images without costly per-scene optimization or 3D annotations.
Approach
Distill offline per-scene optimized EmerNeRFs (dense depth and virtual-camera RGB/depth targets) and 2D foundation model features (CLIP, DINOv2) into a feed-forward two-stage LSS encoder that builds a parameterized sparse hierarchical voxel field rendered via volume rendering.
Key results
- Reconstruction PSNR 30.11 vs EmerNeRF 30.88 on nuScenes, versus 20.67 for SelfOcc, a generalizable baseline.
- Zero-shot transfer to Waymo NOTR then fine-tuning yields PSNR 29.84 and SSIM 0.911, above EmerNeRF's 28.87 and 0.814.
- Zero-shot occupancy mIoU improves from 4.63 to 8.93 with depth distillation from offline NeRFs; F-mIoU is 6.40 vs 5.33 for OccNeRF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add dense depth distillation from offline NeRFs (ablation) | DistillNeRF · nuScenes RGB reconstruction | PSNR: 28.01 to 30.11 | Helped |
| add density complement (ablation) | DistillNeRF · nuScenes RGB reconstruction | PSNR: 22.76 to 28.01 | Helped |
| add decoder (ablation) | DistillNeRF · nuScenes RGB reconstruction | PSNR: 25.76 to 28.01 | Helped |
| use a pretrained 2D encoder (ablation) | DistillNeRF · nuScenes RGB reconstruction | PSNR: 21.35 to 28.01 | Helped |
| use two-stage LSS (ablation) | DistillNeRF · nuScenes RGB reconstruction | PSNR: 27.40 to 28.01 | Helped |
| add virtual-camera distillation (ablation) | DistillNeRF with depth and parameterized space · nuScenes val novel-view synthesis | PSNR: 20.06 -> 20.78 | Helped |
| add virtual-camera distillation (ablation) | DistillNeRF with depth and parameterized space · nuScenes val novel-view synthesis | SSIM: 0.565 -> 0.590 | Helped |
| use parameterized space (ablation) | DistillNeRF with depth distillation · nuScenes val RGB reconstruction | PSNR: 30.11 -> 28.42 | Hurt |
| add virtual-camera distillation (ablation) | DistillNeRF with depth distillation · nuScenes val sparse LiDAR depth | Abs Rel: 0.233 -> 0.223 | Helped |
| add virtual-camera distillation (ablation) | DistillNeRF with depth distillation · nuScenes val dense depth | Abs Rel: 0.235 -> 0.228 | Helped |
| add depth distillation from offline NeRFs (ablation) | DistillNeRF · Occ3D-nuScenes | mIoU: 4.63 to 8.93 | Helped |
| render foundation features directly instead of rendering RGB then running a foundation model | DistillNeRF · Foundation feature reconstruction | CLIP inference time (s): 1.65651 vs. 0.50175 | Helped |
Surprising findings
- Distilling depth from offline NeRFs raised PSNR from 28.01 to 30.11 and also produced the largest occupancy gain, indicating geometry distillation drives both rendering and occupancy.
- The model's rendered DINOv2/CLIP features underperform simply rendering RGB and feeding it to the foundation model on reconstruction PSNR (18.48 vs 21.70 for DINO), though rendering features is about 2-3x faster.
Limitations
- Relies on LiDAR to train offline EmerNeRFs for distillation.
- Sparse voxel representation trades rendering efficiency for dense scene representation.
- Training instabilities and oscillations observed, particularly in depth prediction.
Experimental setup
Datasets: nuScenes Waymo NOTR Occ3D-nuScenes
Benchmarks: nuScenes Waymo NOTR Occ3D-nuScenes
Metrics: PSNR SSIM Abs Rel RMSE delta<1.25 mIoU F-mIoU G-IoU
Baselines: EmerNeRF Single-Frame EmerNeRF SelfOcc UniPAD SimpleOcc OccNeRF
Builds on: EmerNeRF Lift-Splat-Shoot NeRF DINOv2 CLIP
Closest papers by meaning
- Masked Scene Modeling: Narrowing the Gap Between Supervised and Self-Supervised Learning in 3D Scene Understanding (CVPR 2025 · similarity 0.88)
- Boosting Self-Supervision for Single-View Scene Completion via Knowledge Distillation (CVPR 2024 · similarity 0.88)
- Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields (CVPR 2024 · similarity 0.87)
- Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views (AAAI 2024 · similarity 0.87)
- DUNE: Distilling a Universal Encoder from Heterogeneous 2D and 3D Teachers (CVPR 2025 · similarity 0.87)
- From Thousands to Billions: 3D Visual Language Grounding via Render-Supervised Distillation from 2D VLMs (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.