Research map / 3D vision / 3D generation
Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PreWorld, a semi-supervised vision-centric 3D occupancy world model that pre-trains with 2D rendering supervision (RGB, depth, semantic) and fine-tunes with 3D occupancy labels, using an end-to-end state-conditioned MLP forecasting module.
Problem
3D occupancy world models for autonomous driving forecast future occupancy but rely on costly 3D occupancy labels and an indirect pipeline (frozen occupancy model plus tokenizer/autoregressive/decoder) that loses information.
Approach
Replace the tokenize-autoregress-decode forecasting pipeline with a two-MLP state-conditioned module that forecasts future volume features directly, so the whole model trains end-to-end; the preserved future features allow an attribute projection head to render future RGB, depth and semantics for temporal 2D self-supervision before 3D fine-tuning.
Key results
- 3D occupancy prediction mIoU 34.69 with 2D+3D pre-training vs 33.86 for OccFlowNet on Occ3D-nuScenes
- 4D forecasting average mIoU 9.55 vs 8.66 for OccLLaMA-F (3D occupancy supervision)
- Self-supervised PreWorld-S raises 4D forecasting average mIoU from 0.26 (OccWorld-S) to 3.78
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add 2D pre-training before 3D fine-tuning (ablation) | PreWorld · Occ3D-nuScenes 3D occupancy prediction | mIoU (%): 33.95 -> 34.69; +0.74 mIoU | Helped |
| Add 2D pre-training before 3D occupancy forecasting (ablation) | PreWorld · Occ3D-nuScenes 4D occupancy forecasting average | mIoU (%): 9.06 -> 9.55 | Helped |
| Add 2D pre-training before 3D occupancy forecasting (ablation) | PreWorld · Occ3D-nuScenes 4D occupancy forecasting average | IoU (%): 20.88 -> 21.62 | Helped |
| Add ego-state information (ablation) | PreWorld · Occ3D-nuScenes motion planning average | L2 (m): 1.34 -> 0.31 | Helped |
| Add ego-state information (ablation) | PreWorld · Occ3D-nuScenes motion planning average | Collision Rate (%): 1.14 -> 0.54 | Helped |
| Add RGB supervision during pre-training (ablation) | PreWorld · Occ3D-nuScenes 3D occupancy prediction | mIoU (%): 33.95 -> 34.11; (+0.16) | Helped |
| Add depth supervision to RGB pre-training (ablation) | PreWorld · Occ3D-nuScenes 3D occupancy prediction | mIoU (%): 34.11 -> 34.43; (+0.48) | Helped |
| Add semantic supervision to RGB and depth pre-training (ablation) | PreWorld · Occ3D-nuScenes 3D occupancy prediction | mIoU (%): 34.43 -> 34.69; (+0.74) | Helped |
| Increase pre-training data from none to 700 scenes (ablation) | PreWorld fine-tuned on 150 scenes · Occ3D-nuScenes 3D occupancy prediction | mIoU (%): 18.66 -> 25.02; (+6.36) | Helped |
| Add trajectory supervision (ablation) | PreWorld with ego state · Occ3D-nuScenes 4D forecasting average | mIoU (%): 8.85 -> 9.06 | Helped |
| Add trajectory supervision (ablation) | PreWorld with ego state and pre-training · Occ3D-nuScenes 4D forecasting average | mIoU (%): 9.35 -> 9.55 | Helped |
| Add 3D occupancy supervision to trajectory supervision (ablation) | PreWorld motion planning with ego state · Occ3D-nuScenes motion planning average | L2 (m): 0.45 -> 0.31 | Helped |
Surprising findings
- Pre-training improved RayIoU for large static categories even though mIoU declined in some of them, which the authors attribute to mIoU's visible-mask evaluation favoring thicker surfaces
- Ego-state input substantially improves planning L2 in the paper's setup, which the authors attribute to a shortcut effect
Limitations
- Pre-training gains on the full 700-scene fine-tuning set are modest (+0.74 mIoU), and the authors note mIoU may not reflect them well
- Forecasting in the non-ego-state setting is weaker than OccLLaMA on IoU
- Ego-state conditioned planning benefit may stem from shortcut effects, left for future analysis
Experimental setup
Datasets: Occ3D-nuScenes nuScenes
Benchmarks: Occ3D-nuScenes
Metrics: mIoU IoU L2 collision rate RayIoU
Baselines: OccWorld OccLLaMA OccFlowNet RenderOcc SelfOcc OccNeRF SparseOcc UniAD
Builds on: RenderOcc OccWorld BEVStereo FB-OCC NeRF
Closest papers by meaning
- Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving (AAAI 2025 · similarity 0.92)
- UnO: Unsupervised Occupancy Fields for Perception and Forecasting (CVPR 2024 · similarity 0.92)
- GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Prediction (CVPR 2025 · similarity 0.90)
- Achieving Speed-Accuracy Balance in Vision-based 3D Occupancy Prediction via Geometric-Semantic Disentanglement (AAAI 2025 · similarity 0.90)
- Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applications (CVPR 2024 · similarity 0.90)
- COME: Adding Scene-Centric Forecasting Control to Occupancy World Model (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.