Research map / Image generation / Fast diffusion sampling
DetDiffusion: Synergizing Generative and Perceptive Models for Enhanced Data Generation and Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
DetDiffusion fine-tunes Stable Diffusion v1.5 for layout-to-image generation with a perception-aware segmentation loss and detector-derived easy/hard prompt attributes, producing synthetic COCO data that improves detector training.
Problem
Perceptive models need large labeled datasets, and existing layout-to-image diffusion methods neither exploit perceptive-model signals to improve generation nor tailor synthetic data to specific detectors.
Approach
Couples generative and perceptive models in both directions: a segmentation head on UNet features gives a perception-aware loss for better controllability, and detector-derived per-object difficulty tags (easy/hard) are added as prompt tokens so generated data can be steered toward detector-specific hard samples.
Key results
- DetDiffusion easy reaches 31.2 mAP YOLO Score on COCO layout-to-image vs 29.1 for GeoDiffusion; DetDiffusion origin achieves best FID 19.28 vs 20.16.
- Training a Faster R-CNN detector on DetDiffusion synthetic COCO data gives 35.4 mAP (hard strategy) vs 34.5 real-only and 34.8 for GeoDiffusion.
- Adding P.A. loss and P.A. Attr together lowers FID from 20.16 to 19.66 and raises YOLO mAP from 29.1 to 31.2 in ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use perception-aware attributes, assigning [easy] to all objects (ablation) | DetDiffusion · COCO val-set | mAP: 29.1 -> 31.2 | Helped |
| Use perception-aware attributes, assigning [hard] to all objects (ablation) | DetDiffusion · COCO val-set | mAP: 29.8 -> 25.3 | Hurt |
| Add perception-aware attributes (ablation) | DetDiffusion · COCO val-set | mAP: 29.1 -> 30.4 | Helped |
| Add perception-aware loss alongside perception-aware attributes (ablation) | DetDiffusion · COCO val-set | mAP: 30.4 -> 31.2 | Helped |
| Use hard perception-aware attributes for generated training data (ablation) | COCO trainability · COCO | mAP: 34.5 -> 35.4 | Helped |
| Use easy perception-aware attributes for generated training data (ablation) | COCO trainability · COCO | mAP: 34.5 -> 35.2 | Helped |
Surprising findings
- Generating deliberately hard-labeled samples gave larger detector gains than origin or easy samples, suggesting difficulty-controlled synthesis works as stronger augmentation.
Limitations
- Evaluation limited to object detection on COCO at 256x256 for fidelity.
- Gains in detector mAP are modest (about 0.9 mAP over origin in trainability).
- Relies on a pretrained detector to produce easy/hard attributes, so detector choice affects attribute quality.
Experimental setup
Datasets: COCO-Thing-Stuff COCO2017
Benchmarks: COCO-Thing-Stuff val COCO2017 detection
Metrics: FID YOLO Score mAP detector mAP AP50 AP75
Baselines: LostGAN LAMA TwFA Frido LayoutDiffuse LayoutDiffusion ReCo GLIGEN ControlNet GeoDiffusion
Builds on: Latent Diffusion Model GeoDiffusion DatasetDM GLIGEN
Closest papers by meaning
- Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models (ICLR 2025 · similarity 0.88)
- Aligning Generative Denoising with Discriminative Objectives Unleashes Diffusion for Visual Perception (ICLR 2025 · similarity 0.87)
- Scaling Properties of Diffusion Models For Perceptual Tasks (CVPR 2025 · similarity 0.87)
- Auto Cherry-Picker: Learning from High-quality Generative Data Driven by Language (CVPR 2025 · similarity 0.86)
- Generative Perception of Shape and Material from Differential Motion (NeurIPS 2025 · similarity 0.86)
- GeoDiffusion: Text-Prompted Geometric Control for Object Detection Data Generation (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.