Research map / 3D vision / Gaussian splatting
VoxelSplat: Dynamic Gaussian Splatting as an Effective Loss for Occupancy and Flow Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VoxelSplat, a training-only regularization loss that renders decoded dynamic semantic Gaussians into camera views to supervise camera-based occupancy and scene flow prediction, adding no inference cost.
Problem
Camera-based occupancy prediction jointly with scene flow is hard due to occlusion (annotations invisible to cameras) and class/speed imbalance among dynamic objects, and voxel representations model explicit motion poorly.
Approach
Decode sparse semantic 3D Gaussians from voxel features, move dynamic Gaussians by predicted scene flow, and splat them into current and adjacent camera views to get 2D semantic and depth supervision; add speed/class-balanced point sampling. Used only as a training-time loss, so inference is unchanged.
Key results
- FB-OCC + VoxelSplat reaches 36.9 mIoU/37.8 RayIoU-style gains of +3.4 RayIoU and +3.1 mIoU over FB-Occ-flow, with mAVE reduced by 0.202 to 0.303.
- Integrated into BEVDet-Occ, SparseOcc, and FB-Occ, RayIoU improves by 3.6, 2.9, and 3.4 respectively.
- On OpenOccupancy annotations, FB-OCC + Ours improves IoU by 3.48 and mIoU by 3.96 over FB-OCC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add VoxelSplat to FB-Occ-flow (16f) | FB-Occ (16f) · nuScenes val (OpenOcc) | RayIoU: 33.5 -> 36.9 (+3.4) | Helped |
| Add VoxelSplat to FB-Occ-flow (16f) | FB-Occ (16f) · nuScenes val (OpenOcc) | mAVE: 0.505 -> 0.303 (+0.202) | Helped |
| Add VoxelSplat to FB-OCC | FB-OCC · nuScenes val (OpenOccupancy annotations) | SSC mIoU: 18.68 -> 22.64 | Helped |
| Add VoxelSplat to BEVDet-Occ | BEVDet-Occ · nuScenes val (OpenOcc) | RayIoU: 32.0 -> 35.6 (+3.6) | Helped |
| Add VoxelSplat to SparseOcc | SparseOcc · nuScenes val (OpenOcc) | RayIoU: 35.1 -> 38.0 (+2.9) | Helped |
| Use volume rendering supervision (ablation) | BEVDet-Occ-flow (8f) · nuScenes val (OpenOcc) | RayIoU: 32.5 -> 32.9 (+0.4) | Helped |
| Use volume rendering supervision (ablation) | BEVDet-Occ-flow (8f) · nuScenes val (OpenOcc) | mAVE: .545 -> .569 (-.024) | Hurt |
| Use semantic Gaussian splatting supervision (ablation) | BEVDet-Occ-flow (8f) · nuScenes val (OpenOcc) | RayIoU: 32.5 -> 33.6 (+1.1) | Helped |
| Use semantic Gaussian splatting supervision (ablation) | BEVDet-Occ-flow (8f) · nuScenes val (OpenOcc) | mAVE: .545 -> .515 (+.030) | Helped |
| Add 2D ground truth from adjacent frames (ablation) | Gaussian splatting supervision · nuScenes val (OpenOcc) | RayIoU: 33.6 -> 34.1 | Helped |
| Add 2D ground truth from adjacent frames (ablation) | Gaussian splatting supervision · nuScenes val (OpenOcc) | mAVE: 0.515 -> 0.487 | Helped |
| Separate supervision of static and dynamic objects (ablation) | Gaussian splatting with multi-frame supervision · nuScenes val (OpenOcc) | mAVE: 0.487 -> 0.353 | Helped |
Surprising findings
- Applying decomposition and weighted sampling only to the 3D loss, without the 2D rendering loss, causes a significant performance drop.
Limitations
- Still requires ground-truth 3D scene flow despite the self-supervised 2D rendering loss.
- Focused on occupancy and flow prediction; not extended to occupancy forecasting or other tasks.
Experimental setup
Datasets: nuScenes OpenOcc Occ3D OpenOccupancy
Benchmarks: nuScenes validation set (OpenOcc) Occ3D OpenOccupancy
Metrics: RayIoU mIoU mAVE IoU
Baselines: RenderOcc BEVFormer BEVDet-Occ FB-OCC SparseOcc TPVFormer OccFormer SurroundOcc GaussianFormer
Builds on: 3D Gaussian Splatting FB-OCC BEVDet-Occ SparseOcc RenderOcc
Closest papers by meaning
- ODG: Occupancy Prediction Using Dual Gaussians (NeurIPS 2025 · similarity 0.91)
- SceneSplat++: A Large Dataset and Comprehensive Benchmark for Language Gaussian Splatting (NeurIPS 2025 · similarity 0.89)
- STCOcc: Sparse Spatial-Temporal Cascade Renovation for 3D Occupancy and Scene Flow Prediction (CVPR 2025 · similarity 0.89)
- LowRankOcc: Tensor Decomposition and Low-Rank Recovery for Vision-based 3D Semantic Occupancy Prediction (CVPR 2024 · similarity 0.88)
- GaussTR: Foundation Model-Aligned Gaussian Transformer for Self-Supervised 3D Spatial Understanding (CVPR 2025 · similarity 0.88)
- Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.