Research map / 3D vision / Depth and stereo
SelfOcc: Self-Supervised Vision-Based 3D Occupancy Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SelfOcc, a self-supervised vision-based 3D occupancy predictor that learns BEV/TPV representations treated as SDF fields and supervised only by rendered temporal video frames, with an MVS-embedded depth loss.
Problem
Vision-based 3D occupancy prediction normally requires costly 3D occupancy or LiDAR labels; the paper asks whether meaningful 3D occupancy can be learned from video sequences alone, for both monocular and surround-camera settings.
Approach
Lift image features into BEV or TPV 3D representations with deformable attention, decode them into a signed distance field, and render color and depth for previous/future frames via NeuS-style volume rendering; an MVS-embedded loss optimizes SDF-induced rendering weights over multiple depth proposals along the epipolar line instead of bilinear-sampled photometric reprojection.
Key results
- Surround-view self-supervised occupancy on Occ3D-nuScenes reaches 45.01 IoU and 9.30 mIoU (TPV variant).
- On SemanticKITTI, SelfOcc (TPV) gets 21.97 IoU vs 13.84 for SceneRF, a 58.7% relative gain.
- On novel depth synthesis on SemanticKITTI, SelfOcc (TPV) improves Abs Rel to 0.1562 from SceneRF's 0.1681.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SelfOcc (TPV) instead of SceneRF | Self-supervised occupancy prediction · SemanticKITTI | IoU: 13.84 -> 21.97 | Helped |
| Use SelfOcc (TPV) instead of SceneRF | Novel depth synthesis · SemanticKITTI | Abs Rel: 0.1681 -> 0.1562 | Helped |
| Use SelfOcc (TPV) instead of SceneRF | Novel depth synthesis · nuScenes | Abs Rel: 0.7631 -> 0.4003 | Helped |
| Use SelfOcc (TPV) instead of SimpleOcc | Self-supervised surround-view depth estimation · nuScenes | Abs Rel: 0.224 -> 0.215 | Helped |
| Add L_mvs to the density-field architecture (ablation) | SelfOcc · nuScenes occupancy | IoU: 18.90 -> 36.68 | Helped |
| Use an SDF field instead of a density field (ablation) | SelfOcc with L_mvs · nuScenes occupancy | IoU: 36.68 -> 38.43 | Helped |
| Add previous- and next-frame supervision at ratio 2:1:1 (ablation) | SelfOcc · nuScenes occupancy | IoU: 39.67 -> 40.50 | Helped |
| Use current, previous, and next-frame supervision at ratio 1:1:1 (ablation) | SelfOcc · nuScenes occupancy | IoU: 39.67 -> 40.47 | Helped |
| Increase previous- and next-frame supervision to ratio 1:2:2 (ablation) | SelfOcc · nuScenes occupancy | IoU: 39.67 -> 40.43 | Helped |
| Use previous- and next-frame supervision without the current frame (ablation) | SelfOcc · nuScenes occupancy | IoU: 39.67 -> 42.48 | Helped |
| Add L_rgb to L_dep (ablation) | SelfOcc · SemanticKITTI occupancy | IoU: 9.81 -> 24.07 | Helped |
| Add L_H to L_dep + L_rgb (ablation) | SelfOcc · SemanticKITTI occupancy | IoU: 24.07 -> 12.27 | Hurt |
Surprising findings
- Ground-truth poses did not improve SurroundDepth over median scaling, per the authors' retraining with GT poses.
- Sparsity loss makes predictions look closer to ground truth but may not improve IoU under the Occ3D camera-mask protocol.
Limitations
- Semantic predictions depend on the quality of open-vocabulary 2D segmentation (OpenSeeD), leading to weak foreground class performance.
- Temporal supervision trades off occupancy against depth estimation quality.
- Performance on foreground classes remains low compared with supervised methods.
Experimental setup
Datasets: nuScenes Occ3D-nuScenes SemanticKITTI KITTI-2015
Benchmarks: Occ3D-nuScenes SemanticKITTI nuScenes KITTI-2015
Metrics: IoU mIoU Precision Recall Abs Rel Sq Rel RMSE RMSE log delta accuracy
Baselines: SceneRF MonoScene BTS TPVFormer MonoDepth2 SurroundDepth SimpleOcc PixelNeRF MINE VisionNerf
Builds on: NeuS SceneRF BEVFormer TPVFormer Monodepth2
Closest papers by meaning
- OctOcc: High-Resolution 3D Occupancy Prediction with Octree (AAAI 2024 · similarity 0.91)
- ViPOcc: Leveraging Visual Priors from Vision Foundation Models for Single-View 3D Occupancy Prediction (AAAI 2025 · similarity 0.90)
- Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving (ICLR 2025 · similarity 0.89)
- AdaOcc: Adaptive 3D Occupancy Prediction for Embodied Tasks (NeurIPS 2026 · similarity 0.89)
- Achieving Speed-Accuracy Balance in Vision-based 3D Occupancy Prediction via Geometric-Semantic Disentanglement (AAAI 2025 · similarity 0.89)
- SDGOCC: Semantic and Depth-Guided Bird's-Eye View Transformation for 3D Multimodal Occupancy Prediction (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.