Research map / 3D vision / 3D generation
ViPOcc: Leveraging Visual Priors from Vision Foundation Models for Single-View 3D Occupancy Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ViPOcc, a self-supervised single-view 3D occupancy and metric depth framework that uses Depth Anything V2 priors via inverse depth alignment and a Grounded-SAM-guided non-overlapping Gaussian mixture ray sampler.
Problem
Single-view 3D occupancy prediction for autonomous driving with NeRF-based methods lacks instance-level semantic reasoning, temporal photometric consistency, and accurate metric depth; random ray sampling wastes samples and misses instances.
Approach
Couple a metric depth branch to the NeRF occupancy branch: refine Depth Anything V2 pseudo depth via residual inverse-depth fitting, use its depth for temporal photometric warping and depth-rendering consistency, and sample rays with a semantic-guided non-overlapping Gaussian mixture driven by Grounded-SAM instance masks.
Key results
- On KITTI-360 scene occupancy, ViPOcc reaches O_acc^s 0.93 and IE_acc^s/IE_rec^s 0.71/0.69, above BTS (0.92/0.69/0.64) and KYN (0.92/0.70/0.66).
- On KITTI-360 metric depth, ViPOcc achieves Abs Rel 0.097 and RMSE log 0.188, versus 0.103 and 0.194 for BTS.
- On KITTI Raw (Eigen), ViPOcc achieves Abs Rel 0.096 and Sq Rel 0.652, a 5.9% and 11.8% reduction over the previous best.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a metric-depth branch without VFM priors (ablation) | ViPOcc · KITTI-360 scene occupancy | O_acc^s: 0.91 -> 0.91 | No clear effect |
| use VFM pseudo depth directly instead of inverse-depth alignment (ablation) | ViPOcc · KITTI-360 scene occupancy | O_acc^s: 0.91 -> 0.86 | Hurt |
| align VFM pseudo depth in inverse-depth space (ablation) | ViPOcc · KITTI-360 scene occupancy | O_acc^s: 0.91 -> 0.92 | Helped |
| use SNOG rather than random ray sampling (ablation) | ViPOcc · KITTI-360 scene occupancy | IE_acc^s: 0.65 -> 0.67 | Helped |
| use SNOG rather than random ray sampling (ablation) | ViPOcc · KITTI-360 scene occupancy | IE_rec^s: 0.64 -> 0.67 | Helped |
| add temporal alignment loss only (ablation) | ViPOcc · KITTI-360 scene occupancy | O_acc^s: 0.91 -> 0.91 | No clear effect |
| add reconstruction consistency loss only (ablation) | ViPOcc · KITTI-360 scene occupancy | O_acc^s: 0.91 -> 0.89 | Hurt |
| add temporal alignment and reconstruction consistency losses together (ablation) | ViPOcc · KITTI-360 scene occupancy | IE_acc^s: 0.65 -> 0.69 | Helped |
| use ViPOcc rather than KYN for scene occupancy | KITTI-360 scene occupancy · KITTI-360 | IE_rec^s: 0.66 -> 0.69 | Helped |
| use ViPOcc rather than KYN for object occupancy | KITTI-360 object occupancy · KITTI-360 | IE_rec^o: 0.61 -> 0.64 | Helped |
| use ViPOcc rather than BTS for KITTI-360 depth estimation | KITTI-360 depth estimation · KITTI-360 | Abs Rel: 0.103 -> 0.097 | Helped |
| use ViPOcc rather than BTS for KITTI Raw depth estimation | KITTI Raw Eigen split · KITTI Raw | Abs Rel: 0.102 -> 0.096 | Helped |
Surprising findings
- Naively using Depth Anything V2 pseudo depth as a supervisory signal drops scene occupancy accuracy from 0.91 to 0.86 in ablation, and fitting raw depth residuals causes a drastic performance drop.
- Using only the reconstruction consistency loss degrades performance relative to the baseline in ablation.
Limitations
- Depends on pseudo depth from Depth Anything V2 and instance masks from Grounded-SAM, pre-generated offline.
- Performance is limited on distant and small instances, and the framework still relies on NeRF-style volume rendering for occupancy.
- Evaluated only on KITTI-360, KITTI Raw and DDAD, with a modest zero-shot gain.
Experimental setup
Datasets: KITTI-360 KITTI Raw DDAD Cityscapes
Benchmarks: KITTI-360 KITTI Raw (Eigen split) DDAD
Metrics: O_acc^s IE_acc^s IE_rec^s O_acc^o IE_acc^o IE_rec^o Abs Rel Sq Rel RMSE RMSE log
Baselines: Monodepth2 PixelNeRF BTS KYN MVBTS KDBTS SwinDepth Lite-Mono Dynamo-Depth SC-DepthV3
Builds on: BTS Depth Anything V2 Grounded-SAM Monodepth2
Closest papers by meaning
- OctOcc: High-Resolution 3D Occupancy Prediction with Octree (AAAI 2024 · similarity 0.91)
- AdaOcc: Adaptive 3D Occupancy Prediction for Embodied Tasks (NeurIPS 2026 · similarity 0.90)
- RadOcc: Learning Cross-Modality Occupancy Knowledge through Rendering Assisted Distillation (AAAI 2024 · similarity 0.90)
- SelfOcc: Self-Supervised Vision-Based 3D Occupancy Prediction (CVPR 2024 · similarity 0.90)
- Achieving Speed-Accuracy Balance in Vision-based 3D Occupancy Prediction via Geometric-Semantic Disentanglement (AAAI 2025 · similarity 0.90)
- CSV-Occ: Fusing Multi-frame Alignment for Occupancy Prediction with Temporal Cross State Space Model and Central Voting Mechanism (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.