Research map / 3D vision / 3D generation
MagicDrive: Street View Generation with Diverse 3D Geometry Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MagicDrive, a Stable Diffusion-based multi-camera street-view generator conditioned separately on camera poses, text (scene), 3D boxes (cross-attention), and BEV road maps (additive encoder), with cross-view attention for multi-view consistency.
Problem
Street-view image generation with precise 3D geometry control (object height, occlusion, road elevation) and multi-camera consistency, for use as augmentation in perception tasks like BEV segmentation and 3D object detection.
Approach
Encode each condition according to its structure: camera pose and text via scene-level cross-attention, variable-length 3D boxes via cross-attention tokens filtered by visibility, and BEV road maps via an additive trainable encoder branch that performs implicit view transformation; add cross-view attention between adjacent camera views in the UNet.
Key results
- MagicDrive at 224x400 achieves FID 16.20 versus 25.54 for BEVGen and 24.85 for BEVControl on nuScenes validation.
- At 272x736, MagicDrive reaches 31.05 vehicle mIoU and 20.85 mAP / 30.26 NDS with BEVFusion, against an oracle-at-224x400 of 23.54 mAP.
- Adding MagicDrive synthetic data to BEVFusion training improves CAM-only mAP from 32.88 to 35.40 at 1x epochs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use separate 3D bounding-box encoding rather than encoding road and object semantics together in the BEV map (ablation) | MagicDrive · nuScenes val, BEV segmentation | vehicle mIoU | Helped |
| Filter boxes to visible objects in each view (ablation) | MagicDrive · nuScenes val, BEV segmentation | road and vehicle mIoU | Helped |
| Add the box encoder while retaining road and object semantics in the BEV map (ablation) | MagicDrive · nuScenes val, BEV segmentation | segmentation performance | No clear effect |
| Use 1 attending view instead of 2 (ablation) | MagicDrive · Synthetic nuScenes val | FID: 14.46 -> 13.06 | Helped |
| Use 1 attending view instead of 2 (ablation) | MagicDrive · Synthetic nuScenes val | road mIoU: 59.31 -> 58.63 | Hurt |
| Use 1 attending view instead of 2 (ablation) | MagicDrive · Synthetic nuScenes val | vehicle mIoU: 27.13 -> 26.41 | Hurt |
| Use all 5 views instead of 2 for attention (ablation) | MagicDrive · Synthetic nuScenes val | road and vehicle mIoU: 59.31 -> 58.35; 27.13 -> 25.41 | Hurt |
| Use all 5 views instead of 2 for attention (ablation) | MagicDrive · Synthetic nuScenes val | FID: 14.46 -> 14.76 | Hurt |
| Increase classifier-free guidance scale (ablation) | MagicDrive · nuScenes val | FID | Hurt |
| Increase classifier-free guidance scale (ablation) | MagicDrive · nuScenes val, CVT segmentation | vehicle mIoU | Mixed |
| Set the unconditional-inference map to M={0} for CFG (ablation) | MagicDrive · nuScenes val, CVT segmentation | road mIoU | Helped |
| Set the unconditional-inference map to M={0} for CFG (ablation) | MagicDrive · nuScenes val, CVT segmentation | vehicle mIoU | Hurt |
Surprising findings
- At 224x400, MagicDrive-generated images yield 12.30 mAP for 3D detection, below the 23.54 mAP oracle at the same resolution in the paper's own table, suggesting weaker detection fidelity at lower resolution.
- Attending to only one neighboring view gives better FID (13.06) than attending to two views (14.46), but worse multi-camera consistency in mIoU.
Limitations
- Night views are not as dark as real images.
- Cannot generate weather conditions unseen in nuScenes.
- Distant objects are generated less faithfully, and results are evaluated only with synthetic data as augmentation alongside real data.
Experimental setup
Datasets: nuScenes
Benchmarks: nuScenes
Metrics: FID road mIoU vehicle mIoU mAP NDS
Baselines: BEVGen BEVControl
Builds on: ControlNet Stable Diffusion MVDiffusion BEVGen BEVControl
Closest papers by meaning
- StreetCrafter: Street View Synthesis with Controllable Video Diffusion Models (CVPR 2025 · similarity 0.86)
- BEVSpread: Spread Voxel Pooling for Bird's-Eye-View Representation in Vision-based Roadside 3D Object Detection (CVPR 2024 · similarity 0.85)
- DriveGEN: Generalized and Robust 3D Detection in Driving via Controllable Text-to-Image Diffusion Generation (CVPR 2025 · similarity 0.85)
- GeoBEV: Learning Geometric BEV Representation for Multi-view 3D Object Detection (AAAI 2025 · similarity 0.85)
- CorrBEV: Multi-View 3D Object Detection by Correlation Learning with Multi-modal Prototypes (CVPR 2025 · similarity 0.85)
- Effective Multi-sensor Conditioning for Street-view Novel-view Synthesis (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.