Research map / 3D vision / 3D generation
COME: Adding Scene-Centric Forecasting Control to Occupancy World Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COME, a scene-centric forecasting ControlNet that conditions a diffusion-transformer occupancy world model on ego-motion-free future occupancy predictions, improving 3s/8s occupancy generation on Occ3D-nuScenes.
Problem
Occupancy world models for autonomous driving struggle to disentangle ego-vehicle motion (perspective shifts) from scene evolution, leading to spatially inconsistent future occupancy predictions.
Approach
Forecast future occupancy in a scene-centric (fixed) coordinate frame with a simple UNet, encode the transformed predictions with Occ-VAE into scene conditions, and inject them through a trainable ControlNet copy of the first half of a DiT occupancy world model, with a visibility-aware mask on control features.
Key results
- With ground-truth occupancy and ground-truth trajectories (3s), COME achieves 34.23 mIoU vs. DOME's 27.10 (about 26.3% relative gain) and 44.13 IoU vs. 36.36.
- With camera input and predicted trajectories, COME achieves 19.11 average mIoU vs. OccVAR's 11.79.
- With ground-truth occupancy and BEV layouts, COME reaches 39.28 mIoU vs. UniScene-Fore's 31.76 (about 23.7% gain).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add scene-centric forecasting ControlNet to the world model | COME, 3D-Occ with ground-truth trajectory · Occ3D-nuScenes, 1–3s | average mIoU: 27.10 -> 34.23 | Helped |
| Add scene-centric forecasting ControlNet to the small world model (ablation) | COME small model · Occ3D-nuScenes | mIoU: 7.78 -> 32.00 | Helped |
| Unfreeze and train the world model together with ControlNet (ablation) | COME · Occ3D-nuScenes, final-stage training | mIoU: 34.23 -> 28.67 | Hurt |
| Use 10 rather than 5 denoising steps (ablation) | COME · Occ3D-nuScenes, inference | mIoU: 4.95 -> 33.42 | Helped |
| Replace ground-truth occupancy input with stronger EFFOcc occupancy (ablation) | COME · Occ3D-nuScenes | mIoU: 22.26 -> 26.75 | Helped |
| Replace ground-truth pose and yaw with predicted pose and yaw (ablation) | COME · Occ3D-nuScenes | mIoU: 34.23 -> 21.29 | Hurt |
| Align ground-truth occupancy to the predicted future waypoint coordinate (ablation) | COME with predicted pose and yaw · Occ3D-nuScenes | mIoU: 21.29 -> 34.00 | Helped |
| Remove masking from control features (ablation) | COME · Occ3D-nuScenes, 1–3s | average mIoU: 34.23 -> 36.19 | Helped |
| Mask condition features before ControlNet instead of masking control features (ablation) | COME · Occ3D-nuScenes, 1–3s | average mIoU: 34.23 -> 35.37 | Helped |
| Apply random masking dropout during training (ablation) | COME · Occ3D-nuScenes, 1–3s | average mIoU: 34.23 -> 31.97 | Hurt |
Surprising findings
- A vanilla UNet scene-centric forecaster alone reaches 39.12 mIoU, above the full COME model's 34.23 mIoU, though it does poorly on invisible voxels (0.09 mIoU).
- Masking control features on invisible regions lowers the quantitative metrics relative to no masking, even though it gives better qualitative results.
Limitations
- Adding control modules increases computational complexity; real-time use needs optimization.
- Multi-stage training pipeline is not end-to-end and could be streamlined.
- Performance is sensitive to planned trajectory errors when predicted poses are used (21.29 mIoU vs. 34.23 with GT trajectories).
Experimental setup
Datasets: Occ3D-nuScenes nuScenes
Benchmarks: Occ3D-nuScenes
Metrics: mIoU IoU
Baselines: DOME UniScene OccWorld OccLLaMA OccVAR DFIT-OccWorld Occ-LLM RenderWorld Copy&Paste
Builds on: ControlNet DOME OccWorld UNet DiT
Closest papers by meaning
- Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving (AAAI 2025 · similarity 0.92)
- GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Prediction (CVPR 2025 · similarity 0.90)
- Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving (ICLR 2025 · similarity 0.90)
- Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applications (CVPR 2024 · similarity 0.89)
- UnO: Unsupervised Occupancy Fields for Perception and Forecasting (CVPR 2024 · similarity 0.89)
- OccProphet: Pushing the Efficiency Frontier of Camera-Only 4D Occupancy Forecasting with an Observer-Forecaster-Refiner Framework (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.