Research map / Image generation / Text-to-image diffusion

DriveGEN: Generalized and Robust 3D Detection in Driving via Controllable Text-to-Image Diffusion Generation

CVPR 2025 · Image generation › Text-to-image diffusion · approach: Text-driven synthesis and semantic guidance · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DriveGEN, a training-free controllable text-to-image diffusion method that generates OOD driving scenes (snow, fog, rain, night, etc.) with self-prototype and feature-alignment guidance to augment data and improve vision-centric 3D detectors.

Problem

Vision-centric 3D object detectors for autonomous driving degrade under out-of-distribution (OOD) shifts such as weather and corruptions, and existing controllable diffusion augmentation either needs training or loses precise object geometry for tiny objects.

Approach

Training-free controllable T2I generation that extracts PCA principal components of self-attention features, re-weights them by object layouts with a peak function to form self-prototypes, and then guides denoising with semantic-aware feature alignment plus shallow (noisy-latent) feature alignment to preserve object geometry, including for tiny objects.

Key results

• Even with a single augmentation, DriveGEN improves Monoflex by an average of 7.6 mAP across 13 OOD scenarios on KITTI-C.
• On KITTI-C severity 1 Car, Monoflex with DriveGEN (6 scenarios aug.) reaches 37.98 average AP3D versus 19.58-type baselines from FreeControl and 14.09-level baseline numbers (Monoflex avg 26.45 vs 37.98 for DriveGEN 6 scenarios).
• DriveGEN outperforms training-free FreeControl and training-based ControlNet, which degrades severely when augmenting many scenarios.

Measured effects reported in the paper

Surprising findings

• Training-based ControlNet augmentation often degrades detector performance below the unaugmented baseline, especially as more scenarios are added.

Limitations

• Relies on coarse self-attention features that can still miss tiny objects, motivating the shallow alignment component.
• Requires layout (bounding box) annotations of training images to guide generation.

Experimental setup

Datasets: KITTI nuScenes KITTI-C Robo3D

Benchmarks: KITTI-C nuScenes Night nuScenes Rainy Robo3D

Metrics: Mean AP3D|R40

Baselines: Monoflex MonoGround ControlNet FreeControl Color Jitter Brightness

Builds on: Plug-and-play diffusion features (Tumanyan et al.) FreeControl Monoflex classifier-free guidance DDIM inversion

Closest papers by meaning

• CorrBEV: Multi-View 3D Object Detection by Correlation Learning with Multi-modal Prototypes (CVPR 2025 · similarity 0.88)
• SimGen: Simulator-conditioned Driving Scene Generation (NeurIPS 2024 · similarity 0.88)
• GeoDiffusion: Text-Prompted Geometric Control for Object Detection Data Generation (ICLR 2024 · similarity 0.87)
• 3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features (CVPR 2024 · similarity 0.87)
• OpenAD: Open-World Autonomous Driving Benchmark for 3D Object Detection (NeurIPS 2025 · similarity 0.87)
• Achieving Speed-Accuracy Balance in Vision-based 3D Occupancy Prediction via Geometric-Semantic Disentanglement (AAAI 2025 · similarity 0.87)