Research map / Image generation / Text-to-image diffusion
GeoDiffusion: Text-Prompted Geometric Control for Object Detection Data Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GeoDiffusion, which encodes bounding boxes and camera views as text prompts (with discretized location tokens) to fine-tune Stable Diffusion for layout-to-image detection data generation, and shows generated images improve detector training.
Problem
Generating high-quality object detection training data with diffusion models requires precise geometric controls (bounding boxes, camera views), which prior layout-to-image methods handle only with specialized modules and limited flexibility.
Approach
Translate geometric conditions into text prompts by discretizing box corners into location tokens added to the text encoder vocabulary, then fine-tune a pre-trained text-to-image latent diffusion model with a foreground re-weighted loss using area-aware weights.
Key results
- On NuImages at 800x456, GeoDiffusion reaches 10.99 FID and 34.5 mAP for generation fidelity, versus 23.26 FID / 22.6 mAP for ControlNet at 512x512 and 16.68 FID for GLIGEN.
- Augmenting NuImages training with GeoDiffusion images raises Faster R-CNN mAP from 36.9 to 38.3, the only method with consistent gains across most classes.
- With only 10% of real NuImages data, GeoDiffusion augmentation raises mAP from 24.6 to 30.1; on COCO, trainability improves mAP from 37.3 to 38.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use text-prompted geometric control with GeoDiffusion instead of LostGAN | NuImages generation at 800×456 · NuImages val | FID: 21.85 FID | Helped |
| Use text-prompted geometric control with GeoDiffusion instead of ControlNet | NuImages generation at 800×456 · NuImages val | mAP: +11.9 mAP | Helped |
| Add GeoDiffusion-generated images to real detector-training data | Faster R-CNN on NuImages · NuImages val | mAP: 36.9 -> 38.3 (+1.4) | Helped |
| Add GeoDiffusion-generated images to real detector-training data | Faster R-CNN on COCO · COCO val | mAP: 37.3 -> 38.4 | Helped |
| Use GeoDiffusion for COCO inpainting instead of Stable Diffusion | COCO inpainting · COCO val | mAP: 17.6 -> 19.0 (+1.4) | Helped |
| Use a finer location grid (increase grid size) (ablation) | GeoDiffusion · NuImages val, 800×456 | mAP: 20.8 -> 21.4 -> 23.7 | Helped |
| Use the pretrained text encoder instead of random initialization (ablation) | GeoDiffusion · NuImages val, 800×456 | mAP: 9.5 -> 23.7 | Helped |
| Add constant foreground re-weighting with w=2.0 (ablation) | GeoDiffusion · NuImages val, 800×456 | mAP: 23.7 -> 27.1 (+3.4) | Helped |
| Add area re-weighting with p=0.2 to constant foreground re-weighting (ablation) | GeoDiffusion · NuImages val, 800×456 | mAP: 27.1 -> 30.1 | Helped |
| Add area re-weighting with p=0.2 to constant foreground re-weighting (ablation) | GeoDiffusion · NuImages val, 800×456 | FID: 11.77 -> 11.99 | Hurt |
| Normalize the foreground re-weighting mask (ablation) | GeoDiffusion · NuImages val, 800×456 | FID: 14.76 -> 11.99 | Helped |
| Add camera-view text conditioning (ablation) | GeoDiffusion · NuImages val, 800×456 | mAP: 31.0 -> 34.5 | Helped |
Surprising findings
- Initializing the text encoder from pre-trained weights helps substantially: random text-encoder initialization raises FID from 11.63 to 16.45 and drops mAP from 23.7 to 9.5 in ablation.
Limitations
- Generated images only serve as augmentation alongside real data; training detectors solely on synthetic images is not explored.
- Struggles with unseen classes during fine-tuning (catastrophic forgetting) and with high-variance or occluded objects such as pedestrians and cars.
- Text-based location encoding does not support dense pixel-level control such as mask-to-image generation.
Experimental setup
Datasets: NuImages COCO COCO-Stuff LVIS
Benchmarks: NuImages COCO COCO-Stuff
Metrics: FID COCO-style mAP AP50 AP75 YOLO score
Baselines: LostGAN LAMA Taming ReCo GLIGEN ControlNet LayoutDiffuse Frido CAL2IM TwFA Stable Diffusion
Builds on: Stable Diffusion latent diffusion models ReCo Pix2Seq-style coordinate discretization Faster R-CNN
Closest papers by meaning
- DriveGEN: Generalized and Robust 3D Detection in Driving via Controllable Text-to-Image Diffusion Generation (CVPR 2025 · similarity 0.87)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.87)
- GeoDiT: A Diffusion-based Vision-Language Model for Geospatial Understanding (CVPR 2026 · similarity 0.87)
- SemCity: Semantic Scene Generation with Triplane Diffusion (CVPR 2024 · similarity 0.86)
- Spatial Transport Optimization by Repositioning Attention Map for Training-Free Text-to-Image Synthesis (CVPR 2025 · similarity 0.86)
- Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.