Research map / Video / Video generation
RLGF: Reinforcement Learning with Geometric Feedback for Autonomous Driving Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RLGF, which fine-tunes video diffusion models (MagicDrive-V2, DiVE) with LoRA using rewards from latent-space geometry and occupancy perception models over a sliding window of denoising steps, improving geometric fidelity and 3D detection mAP on nuScenes.
Problem
Video generation models for autonomous driving produce visually realistic videos with subtle geometric distortions (vanishing point, lane topology, depth), which degrade downstream 3D object detection when used as synthetic training data.
Approach
Apply RL-style reward feedback from frozen latent-space perception models (vanishing point, lane, depth, and occupancy) to intermediate noisy latents within a randomly placed sliding window of denoising steps, with a micro-decode module to avoid full VAE decoding, and fine-tune only LoRA adapters.
Key results
- DiVE+RLGF raises BEVFusion mAP from 25.75 to 31.42 and NDS from 33.61 to 36.07, against 35.53 mAP for real data.
- DiVE+RLGF reduces depth RMSE from 1.822 to 0.772 and VP error from 0.086 to 0.068, and raises lane F1 from 0.792 to 0.879.
- MagicDrive-v2+RLGF improves mAP from 18.95 to 23.21 while FVD stays near baseline (101.2 to 99.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply RLGF to DiVE | DiVE · nuScenes val | mAP: 25.75 to 31.42 | Helped |
| Apply RLGF to DiVE | DiVE · nuScenes val | VP error: 0.086 → 0.068 | Helped |
| Apply RLGF to DiVE | DiVE · nuScenes val | Lane F1: 0.792 →0.879 | Helped |
| Apply RLGF to DiVE | DiVE · nuScenes val | Depth RMSE: 1.822 → 0.772 | Helped |
| Apply RLGF to DiVE | DiVE · nuScenes val | FVD: 68.4 to 67.6 | Helped |
| Apply RLGF to MagicDrive-v2 | MagicDrive-v2 · nuScenes val | mAP: 18.95 → 23.21 | Helped |
| Add VP reward (ablation) | DiVE · nuScenes val | mAP: 25.75 to 26.31 | Helped |
| Add lane reward to VP reward (ablation) | DiVE · nuScenes val | mAP: 26.31 to 26.93 | Helped |
| Add depth reward to VP and lane rewards (ablation) | DiVE · nuScenes val | mAP: 26.93 to 27.12 | Helped |
| Use all five HGA reward components instead of point-line-plane rewards alone (ablation) | DiVE · nuScenes val | mAP: 27.12 to 31.42 | Helped |
| Use occupancy alignment and IoU rewards only (ablation) | DiVE · nuScenes val | mAP: 25.75 to 28.06 | Helped |
| Set window size to 3 instead of 5 (ablation) | RLGF on DiVE · nuScenes val | mAP: 31.42 to 30.89 | Hurt |
Surprising findings
- A detector-based reward (mAP 26.51) gives only a small gain over the DiVE baseline (25.75), far below the full HGR reward (31.42).
- 2D detection on DiVE synthetic data (mAP 43.8) is close to real data (44.7), suggesting the gap is mainly in 3D geometry.
Limitations
- Performance depends on accuracy of pseudo-labeled frozen perception models, which may propagate bias into rewards.
- RL fine-tuning is more computationally expensive than standard diffusion training; reward weights are manually tuned.
- Evaluated only on nuScenes; generalization to other datasets, weather, and night scenes is untested.
Experimental setup
Datasets: nuScenes Occ3D-nuScenes
Benchmarks: nuScenes Occ3D-nuScenes
Metrics: mAP NDS FVD VP NormDist Lane F1 Depth RMSE mIoU
Baselines: MagicDrive-V2 DiVE Panacea Drive-WM DetectorReward
Builds on: DiVE MagicDrive-V2 DINOv2 Depth Anything V2 BEVFusion
Closest papers by meaning
- DriveGEN: Generalized and Robust 3D Detection in Driving via Controllable Text-to-Image Diffusion Generation (CVPR 2025 · similarity 0.87)
- Glad: A Streaming Scene Generator for Autonomous Driving (ICLR 2025 · similarity 0.86)
- RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Driving (CVPR 2026 · similarity 0.86)
- X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability (NeurIPS 2025 · similarity 0.86)
- Achieving Speed-Accuracy Balance in Vision-based 3D Occupancy Prediction via Geometric-Semantic Disentanglement (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.