Research map / Image generation / Text-to-image diffusion
Slot-Guided Adaptation of Pre-trained Diffusion Models for Object-Centric Learning and Compositional Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SlotAdapt, which attaches slot-conditioned adapter cross-attention layers to a frozen pretrained Stable Diffusion UNet, plus a guidance loss aligning adapter and slot attention masks, for object-centric learning and compositional generation on real images.
Problem
Slot-based object-centric learning with pretrained diffusion decoders suffers from text-trained cross-attention biases, limiting object discovery and compositional generation on complex real-world images.
Approach
Insert adapter cross-attention layers that condition the frozen pretrained diffusion UNet on slots (plus a pooled register token) instead of text cross-attention, and align adapter attention masks with slot attention masks via a BCE guidance loss without external supervision.
Key results
- On COCO, Ours + Guidance reaches instance mBO 35.1 and class mBO 39.2 versus 27.0/30.5 for LSD and 31.0/35.0 for SlotDiffusion, with FG-ARI 41.4 vs 37.2 for SlotDiffusion.
- On COCO compositional/reconstruction generation, FID 40.568 vs 64.213 for SlotDiffusion (compositional) and 10.857 vs 19.448 for SlotDiffusion (reconstruction).
- On VOC, SlotAdapt achieves FG-ARI 29.6 (best) and instance mBO 51.5, but SlotDiffusion remains best on class mBO (55.3 vs 52.0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SlotAdapt instead of LSD | object discovery and segmentation · MOVi-E | FG-ARI: 52.17 -> 56.45 | Helped |
| Use SlotAdapt instead of SlotDiffusion | object segmentation · COCO | mBOᶦ: 31.0 -> 35.1 | Helped |
| Use SlotAdapt instead of SlotDiffusion | compositional generation · COCO | FID: 64.213 -> 40.568 | Helped |
| Add joint attention guidance (ablation) | SlotAdapt · COCO | mBOᶦ: 31.5 -> 35.1 | Helped |
| Add joint attention guidance (ablation) | SlotAdapt · VOC | mBOᶜ: 52.0 -> 51.9 | Hurt |
| Guide slot attention only instead of using joint guidance (ablation) | SlotAdapt · COCO | mBOᶦ: 35.1 -> 33.4 | Hurt |
| Guide diffusion attention only instead of using joint guidance (ablation) | SlotAdapt · COCO | mBOᶦ: 35.1 -> 31.2 | Hurt |
| Use multiplication guidance instead of joint guidance (ablation) | SlotAdapt · COCO | FG-ARI: 41.4 -> 43.3 | Helped |
| Increase slot count from 7 to 80 (ablation) | SlotAdapt · COCO | FG-ARI: 41.4 -> 18.1 | Hurt |
| Delay guidance loss until after 40K iterations instead of starting at 0 (ablation) | SlotAdapt · COCO | FG-ARI: 37.85 -> 41.4 | Helped |
| Use a dedicated additional global slot instead of slot averaging (ablation) | SlotAdapt · COCO | FID: 10.857 -> 11.212 | Hurt |
| Use BOQ-SA instead of Slot Attention (ablation) | SlotAdapt · COCO | mIoUᶦ: 31.7 -> 32.551 | Mixed |
Surprising findings
- Adding guidance on COCO with multiplication guidance gives best FG-ARI (43.3) but joint guidance is best on mBO/mIoU, showing FG-ARI is not sufficient for generative tasks.
- Using 80 slots instead of 7 degraded COCO FG-ARI from 41.4 to 18.1 and sharply worsened FID.
Limitations
- Edited or reconstructed images may show slight changes from the source image.
- Under- and over-segmentation issues remain.
- Pretrained diffusion models trained on real data may be less effective on synthetic datasets such as CLEVR and CLEVRTex.
Experimental setup
Datasets: MOVi-E PASCAL VOC MS COCO
Benchmarks: MOVi-E PASCAL VOC MS COCO
Metrics: FG-ARI mBO mIoU FID KID category accuracy position MSE 3D bounding box MSE
Baselines: SLATE SLATE+ Latent Slot Diffusion Slot Attention DINOSAUR SlotDiffusion GLASS SPOT
Builds on: Slot Attention Latent Slot Diffusion SlotDiffusion T2I-Adapter Latent Diffusion Models DINOv2
Closest papers by meaning
- GLASS: Guided Latent Slot Diffusion for Object-Centric Learning (CVPR 2025 · similarity 0.93)
- Grounded Object-Centric Learning (ICLR 2024 · similarity 0.90)
- Adaptive Slot Attention: Object Discovery with Dynamic Slot Number (CVPR 2024 · similarity 0.89)
- MetaSlot: Break Through the Fixed Number of Slots in Object-Centric Learning (NeurIPS 2025 · similarity 0.88)
- Learning to Compose: Improving Object Centric Learning by Injecting Compositionality (ICLR 2024 · similarity 0.88)
- Identifiable Object-Centric Representation Learning via Probabilistic Slot Attention (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.