Research map / Image generation / Text-to-image diffusion
GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GrounDiT, a training-free bounding-box grounding method for DiT text-to-image generation that cultivates noisy patches per box via joint denoising (semantic sharing) and transplants them into the main latent.
Problem
Training-free spatial grounding of text-to-image diffusion models with bounding boxes; prior loss-guided methods lack precise per-box control, leading to missing objects and box misalignment.
Approach
Discovers "semantic sharing" in DiT: jointly denoising a small patch with a generatable-size image makes them semantically correlated even across different resolutions. Each bounding box gets its own object branch that denoises a noisy object image jointly with the cropped noisy patch, and the resulting patch is transplanted back into the main image at every timestep, after a global cross-attentio
Key results
- On HRS spatial accuracy, GrounDiT reaches 45.01% vs 30.14% for R&B and 37.13% for PixArt-R&B.
- On HRS-Spatial mIoU, GrounDiT scores 0.372 vs 0.326 for R&B.
- Inference time with six boxes is 1.41x that of R&B (55.30s vs 39.15s per the appendix table).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the Global Update to PixArt-α (ablation) | PixArt-α · HRS | spatial accuracy (%): 17.86% to 37.13% | Helped |
| Add Local Update with noisy patch transplantation to PixArt-R&B (ablation) | PixArt-α with Global Update (PixArt-R&B) · HRS | spatial accuracy (%): 37.13% to 45.01% | Helped |
| Add Local Update with noisy patch transplantation to PixArt-R&B (ablation) | PixArt-α with Global Update (PixArt-R&B) · HRS | size accuracy (%) | Helped |
| Add Local Update with noisy patch transplantation to PixArt-R&B (ablation) | PixArt-α with Global Update (PixArt-R&B) · HRS | color accuracy (%): +6.60% | Helped |
| Add Local Update with noisy patch transplantation to PixArt-R&B (ablation) | PixArt-α with Global Update (PixArt-R&B) · DrawBench | spatial accuracy (%): 60.00 to 60.00 | No clear effect |
| Add Local Update with noisy patch transplantation to PixArt-R&B (ablation) | PixArt-α with Global Update (PixArt-R&B) · HRS | CLIP score: 33.49 to 33.63 | Helped |
| Add Local Update with noisy patch transplantation to PixArt-R&B (ablation) | PixArt-α with Global Update (PixArt-R&B) · HRS | ImageReward: 0.28 to 0.44 | Helped |
| Add Local Update with noisy patch transplantation to PixArt-R&B (ablation) | PixArt-α with Global Update (PixArt-R&B) · HRS | PickScore: 0.52 to 0.48 | Hurt |
| Add Local Update with noisy patch transplantation to PixArt-R&B (ablation) | PixArt-α with Global Update (PixArt-R&B) · Subset of MS-COCO-2014 | mIoU: 0.418 to 0.432 | Helped |
| Add Local Update with noisy patch transplantation to PixArt-R&B (ablation) | PixArt-α with Global Update (PixArt-R&B) · HRS-Spatial | mIoU: 0.334 to 0.372 | Helped |
| Add Local Update with noisy patch transplantation to PixArt-R&B (ablation) | PixArt-α with Global Update (PixArt-R&B) · Custom Dataset | mIoU: 0.206 to 0.250 | Helped |
| Increase the proportion of denoising steps using joint denoising (ablation) | DiT joint denoising · Generated image pairs at equal and different resolutions | LPIPS | Helped |
Limitations
- Increased computation time with one object branch per bounding box.
- Generatable-resolution constraint of DiT limits object patch sizes.
Experimental setup
Datasets: HRS DrawBench MS-COCO-2014 subset LayoutGPT-generated custom layouts
Benchmarks: HRS DrawBench MS-COCO-2014 subset HRS-Spatial mIoU
Metrics: spatial accuracy size accuracy color accuracy mIoU CLIP score ImageReward PickScore LPIPS inference time
Baselines: R&B PixArt-R&B BoxDiff Attention-Refocusing Layout-Guidance Stable Diffusion PixArt-alpha
Builds on: R&B PixArt-alpha Layout-Guidance DPM-Solver
Closest papers by meaning
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.88)
- PixelDiT2: Representation-Grounded Pixel Diffusion Transformers (NeurIPS 2026 · similarity 0.88)
- SparseDiT: Token Sparsification for Efficient Diffusion Transformer (NeurIPS 2025 · similarity 0.87)
- Scaling Diffusion Transformers Efficiently via $\mu$P (NeurIPS 2025 · similarity 0.87)
- Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency (ICML 2025 · similarity 0.87)
- Rare Text Semantics Were Always There in Your Diffusion Transformer (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.