Research map / 3D vision / 3D generation
DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DynamicCity, a VAE-plus-DiT framework that encodes 4D occupancy sequences into HexPlanes with a Projection Module and Expansion & Squeeze decoder, then generates them with a DiT using Padded Rollout tokenization.
Problem
Generating large-scale, long-sequence dynamic 4D semantic occupancy scenes for urban driving, which prior methods (mostly static or single-frame) do not handle well.
Approach
Represent 4D occupancy as a compact HexPlane learned by a VAE with transformer-based projection (instead of averaging) and parallel expansion-and-squeeze decoding, then run a DiT over a padded square arrangement of the six planes so that spatial and temporal token relations are preserved; conditions (command, trajectory, layout, HexPlane history) are injected via adaLN-Zero and cross-attention.
Key results
- VAE reconstruction mIoU on CarlaSC (16 frames) reaches 74.22% vs OccSora 33.40%, and the Projection Module adds up to 12.56% mIoU over average pooling.
- Generation on CarlaSC (16 frames) improves 2D FID from 28.55 to 12.94 and 3D FID from 1559 to 354.2 relative to OccSora.
- User study on 840 ratings gives DynamicCity overall quality 4.03 vs OccSora 2.21.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the DynamicCity VAE instead of OccSora | 4D occupancy reconstruction · CarlaSC, 128×128×8, 16 frames | mIoU: 33.40% -> 74.22% | Helped |
| Use the DynamicCity VAE instead of OccSora | 4D occupancy reconstruction · Occ3D-Waymo, 16 frames | mIoU: 36.38% -> 68.18% | Helped |
| Use the DynamicCity VAE instead of OccSora | 4D occupancy reconstruction · Occ3D-nuScenes, 16 frames, 11 classes | mIoU: 13.70% -> 56.93% | Helped |
| Use DynamicCity instead of OccSora | 4D generation · CarlaSC, 16 frames, 2D | FID: 28.55 -> 12.94 | Helped |
| Use DynamicCity instead of OccSora | 4D generation · Occ3D-Waymo, 16 frames, 3D | FID: 3140 -> 1806 | Helped |
| Add the Projection Module instead of Average Pooling (ablation) | DynamicCity VAE · CarlaSC VAE, Query decoder | mIoU: 60.97% -> 68.73% | Helped |
| Add the Projection Module instead of Average Pooling (ablation) | DynamicCity VAE · Occ3D-Waymo VAE, ESS decoder | mIoU: 55.72% -> 62.57% | Helped |
| Add ESS instead of querying each point individually (ablation) | DynamicCity VAE with Projection encoder · CarlaSC VAE | mIoU: 68.73% -> 74.22% | Helped |
| Add ESS instead of querying each point individually (ablation) | DynamicCity VAE with Projection encoder · Occ3D-Waymo VAE | mIoU: 61.93% -> 62.57% | Helped |
| Replace Padded Rollout with Direct Unfold (ablation) | DynamicCity DiT · CarlaSC generation, 2D | FID: 10.96 -> 205.0 | Hurt |
| Replace Padded Rollout with Vertical Concatenation (ablation) | DynamicCity DiT · CarlaSC generation, 2D | FID: 10.96 -> 12.79 | Hurt |
| Increase HexPlane downsampling rates from 2,2,2,2 to 2,4,4,2 (ablation) | DynamicCity VAE · CarlaSC | mIoU: 74.22% -> 65.15% | Hurt |
Surprising findings
- The VAE training and inference VRAM on Occ3D-Waymo is higher than OccSora (24.90G vs 10.05G training), and inference is slower on CarlaSC (0.41s vs 0.21s), despite better quality.
- Sparse convolution was judged to offer only limited efficiency gains because convolution is a small part of the VAE.
Limitations
- Very long autoregressive generation may become less stable.
- Decoder cannot use sparse convolution, and the VAE is heavier in memory and slower at inference than OccSora on some settings.
- Evaluation of 2D/3D generation metrics relies on rendered images and a proxy 3D encoder rather than standard metrics.
Experimental setup
Datasets: Occ3D-Waymo Occ3D-nuScenes CarlaSC
Benchmarks: CarlaSC Occ3D-Waymo Occ3D-nuScenes
Metrics: mIoU IS FID KID precision recall IoU
Baselines: OccSora OccWorld
Builds on: HexPlane DiT OccSora SemCity Lovász-softmax loss
Closest papers by meaning
- Dynamic 3D Gaussian Fields for Urban Areas (NeurIPS 2024 · similarity 0.88)
- SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency (ICLR 2025 · similarity 0.87)
- Diffusion4D: Fast Spatial-temporal Consistent 4D generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.86)
- UniScene: Unified Occupancy-centric Driving Scene Generation (CVPR 2025 · similarity 0.86)
- Multi-Level Neural Scene Graphs for Dynamic Urban Environments (CVPR 2024 · similarity 0.86)
- Flux4D: Flow-based Unsupervised 4D Reconstruction (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.