Research map / Point clouds / Point cloud classification
X-Drive: Cross-modality Consistent Multi-Sensor Data Synthesis for Driving Scenarios
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes X-Drive, a dual-branch latent diffusion model that jointly generates nuScenes LiDAR range images and multi-view camera images with epipolar-line cross-modality conditioning, plus text and 3D box controls.
Problem
Single-modality driving data generators synthesize LiDAR or camera data separately, causing cross-modality misalignment; joint, consistent synthesis of point clouds and multi-view images is under-explored.
Approach
Two latent diffusion branches (range-image LiDAR and multi-view camera) condition each other locally via epipolar-line cross-attention: camera features are sampled along LiDAR rays and LiDAR features along camera-pixel rays, so the model learns correspondence without explicit depth.
Key results
- Joint generation achieves FID 17.37 for images and MMD 1.2e-4 / JSD 0.052 for point clouds, with DAS 1.69 vs 2.32 for MagicDrive+RangeLDM.
- Camera-to-LiDAR conditional generation reaches MMD 1.0e-4 and JSD 0.052, beating RangeLDM (MMD 1.9e-4, JSD 0.054).
- Object-level control on SparseFusion detection gives mAP 65.4 / NDS 69.6, slightly above MagicDrive (65.2 / 69.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use X-Drive instead of MagicDrive for LiDAR-to-camera generation | X-Drive · nuScenes val, LiDAR-to-camera | FID: 16.20 -> 16.01 | Helped |
| Use X-Drive instead of MagicDrive for LiDAR-to-camera generation | X-Drive · nuScenes val, LiDAR-to-camera | VSC: 0.633 -> 0.721 | Helped |
| Use X-Drive instead of RangeLDM for camera-to-LiDAR generation | X-Drive · nuScenes val, camera-to-LiDAR | MMD: $1.9\times 10^{-4}$ -> $\mathbf{1.0\times 10^{-4}}$ | Helped |
| Use X-Drive instead of RangeLDM for camera-to-LiDAR generation | X-Drive · nuScenes val, camera-to-LiDAR | JSD: 0.054 -> 0.052 | Helped |
| Use X-Drive instead of MagicDrive + RangeLDM | joint camera and LiDAR generation · nuScenes val, joint camera and LiDAR generation | DAS: 2.32 -> 1.69 | Helped |
| Use X-Drive instead of MagicDrive | joint camera and LiDAR generation · nuScenes val, joint camera and LiDAR generation | FID: 16.20 -> 17.37 | Hurt |
| Use X-Drive instead of MagicDrive | joint camera and LiDAR generation · nuScenes val, joint camera and LiDAR generation | VSC: 0.633 -> 0.675 | Helped |
| Use X-Drive instead of MagicDrive for object-level control | X-Drive · nuScenes val, object-level control | mAP: 65.2 -> 65.4 | Helped |
| Use X-Drive instead of MagicDrive for object-level control | X-Drive · nuScenes val, object-level control | NDS: 69.2 -> 69.6 | Helped |
| Add the cross-modality condition module (ablation) | X-Drive · nuScenes, joint camera and LiDAR generation | cross-modality alignment | Helped |
| Replace averaging along epipolar lines with cross-attention (ablation) | X-Drive cross-modality condition module · nuScenes, joint camera and LiDAR generation | point-cloud quality | Helped |
| Add text and bounding-box input conditions (ablation) | X-Drive · nuScenes, point-cloud and image generation | data quality | Helped |
Surprising findings
- The joint model's FID (17.37) is worse than MagicDrive alone (16.20), attributed to lacking map conditions.
Limitations
- Limited to single-frame generation without temporal modeling.
- Cannot synthesize rare scenarios such as snowy weather due to nuScenes distribution.
- No map conditioning, which hurts FID relative to MagicDrive.
Experimental setup
Datasets: nuScenes
Benchmarks: nuScenes
Metrics: FID VSC MMD JSD DAS mAP NDS
Baselines: BEVControl BEVGen MagicDrive LiDAR VAE LiDARGen RangeLDM MagicDrive + RangeLDM
Builds on: Latent Diffusion Models MagicDrive RangeLDM LiDARGen Stable Diffusion
Closest papers by meaning
- Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency (NeurIPS 2025 · similarity 0.87)
- Effective Multi-sensor Conditioning for Street-view Novel-view Synthesis (NeurIPS 2026 · similarity 0.87)
- X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability (NeurIPS 2025 · similarity 0.86)
- Scaling Diffusion Models to Real-World 3D LiDAR Scene Completion (CVPR 2024 · similarity 0.85)
- StreetCrafter: Street View Synthesis with Controllable Video Diffusion Models (CVPR 2025 · similarity 0.85)
- LiT: Unifying LiDAR "Languages" with LiDAR Translator (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.