Research map / 3D vision / 3D generation
GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GaussianWorld, a 3D Gaussian world model that reformulates streaming vision-centric 3D occupancy prediction as conditional 4D occupancy forecasting, evolving previous-frame Gaussians with ego alignment, dynamic motion, and new-area completion.
Problem
Vision-centric 3D occupancy prediction with temporal inputs usually fuses past multi-frame features, ignoring scene continuity and the strong prior of scene evolution, which adds computation and limits performance.
Approach
Treat perception as a world model conditioned on current observations: propagate explicit 3D semantic Gaussians from the previous frame through ego-motion alignment, dynamic-Gaussian motion updates, and random Gaussian completion of newly observed areas, then refine them with unified Gaussian world layers, streaming frame by frame.
Key results
- GaussianWorld reaches 21.78 mIoU and 32.75 IoU on nuScenes validation, beating the single-frame GaussianFormer-B (19.73 mIoU, 30.68 IoU) by about 2 points.
- GaussianWorld also outperforms the temporal-fusion GaussianFormer-T (20.42 mIoU, 31.34 IoU) while running at 228 ms and 7030 M memory versus 382 ms and 10019 M for perspective-view fusion.
- Ablations show removing ego motion alignment drops mIoU by 3.0 points, and omitting completion of newly observed areas causes training collapse.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GaussianWorld instead of single-frame GaussianFormer-B | GaussianFormer-B · nuScenes validation set | mIoU: 19.73 -> 21.78 | Helped |
| use GaussianWorld instead of GaussianFormer-T temporal fusion | GaussianFormer-T · nuScenes validation set | mIoU: 20.42 -> 21.78 | Helped |
| replace single-frame inference with GaussianWorld streaming prediction | GaussianFormer-B · nuScenes validation set | Latency: 225 ms -> 228 ms | No clear effect |
| omit ego-motion alignment (ablation) | GaussianWorld · nuScenes validation set | mIoU: 3.0% decrease | Hurt |
| omit local movements of dynamic objects (ablation) | GaussianWorld · nuScenes validation set | mIoU | Hurt |
| omit completion of newly observed areas (ablation) | GaussianWorld · nuScenes validation set | mIoU | Hurt |
| use schedule A instead of schedule E (ablation) | GaussianWorld · nuScenes validation set | mIoU: 20.24 -> 19.77 | Hurt |
| use mean rather than summed gradient accumulation (ablation) | GaussianWorld streaming training · nuScenes validation set | mIoU: 19.77 -> 19.81 | No clear effect |
| use the shorter epoch schedule with no gradient accumulation (ablation) | GaussianWorld streaming training · nuScenes validation set | mIoU: 20.24 -> 18.63 | Hurt |
| train only at sequence length 38 without probabilistic modeling (ablation) | GaussianWorld streaming training · nuScenes validation set | mIoU: 20.24 -> 19.82 | Hurt |
| use probabilistic modeling with gradually increased sequence lengths (ablation) | GaussianWorld streaming training · nuScenes validation set | mIoU: 19.82 -> 20.24 | Helped |
Surprising findings
- Without scene completion of newly observed areas, training collapses as the ego vehicle moves and all Gaussians fall outside the perception range.
- Performance drops slightly beyond around 20 streaming frames, which the authors attribute to sparsity at scene edges in the LiDAR-fused annotations.
Limitations
- Cannot achieve full cross-frame consistency for static scenes due to imperfect disentanglement of dynamic and static elements.
- Performance drops for very long streaming sequences, attributed to annotation sparsity at scene edges.
- Using more refinement layers improves dynamics modeling but harms static-element consistency.
Experimental setup
Datasets: nuScenes SurroundOcc occupancy annotations
Benchmarks: nuScenes occupancy validation set (SurroundOcc labels)
Metrics: IoU mIoU latency memory
Baselines: MonoScene Atlas BEVFormer TPVFormer OccFormer GaussianFormer SurroundOcc GaussianFormer-T
Builds on: GaussianFormer FCOS3D SurroundOcc lovasz-softmax loss
Closest papers by meaning
- GaussianFormer-2: Probabilistic Gaussian Superposition for Efficient 3D Occupancy Prediction (CVPR 2025 · similarity 0.92)
- ODG: Occupancy Prediction Using Dual Gaussians (NeurIPS 2025 · similarity 0.92)
- Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applications (CVPR 2024 · similarity 0.90)
- Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving (ICLR 2025 · similarity 0.90)
- COME: Adding Scene-Centric Forecasting Control to Occupancy World Model (NeurIPS 2025 · similarity 0.90)
- UnO: Unsupervised Occupancy Fields for Perception and Forecasting (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.