Research map / Image generation / Autoregressive and latent generation
Compositional Scene Understanding through Inverse Generative Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames scene understanding as inverse generative modeling: composes diffusion/EBM concept-conditioned models and infers concept parameters that best explain an image, generalizing to more objects and new domains.
Problem
Scene understanding (object discovery, multi-attribute classification, multi-object perception) by discriminative models fails to generalize to test scenes that are more complex or distributionally shifted than training data.
Approach
Build a visual generative model compositionally as a product of concept-conditioned diffusion/EBM terms (summing denoising functions), then infer the concept parameters that minimize denoising error on a given image, using multiple random initializations and SGD for continuous concepts and enumeration or gradient relaxation for discrete ones.
Key results
- On CLEVR out-of-distribution (6-8 objects), the method reaches 85.3% perception rate vs 59.0% for DINOSAUR and 58.7% for Generative Classifier.
- On CelebA male faces (OOD), the method reaches 65.6% accuracy vs 62.2% for ResNet-50.
- Zero-shot multi-object perception with pretrained Stable Diffusion reaches 87.3% accuracy vs 70.4% for Diffusion Classifier on 71 real images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use compositional IGM instead of the strongest listed baseline | CLEVR object discovery · CLEVR, 3-5 and 6-8 objects | perception rate: 82.5% -> 94.7%; 59.0% -> 85.3% | Helped |
| Add multiple random initializations (ablation) | IGM object discovery · CLEVR, 3-5 objects | perception rate: 72.8% -> 94.7% | Helped |
| Add multiple random initializations (ablation) | IGM object discovery · CLEVR, 6-8 objects | perception rate: 68.0% -> 85.3% | Helped |
| Use 5 random initializations instead of 1 (ablation) | IGM object discovery · CLEVR, 3-5 objects | perception rate: 72.8% -> 89.6% | Helped |
| Use 10 random initializations instead of 1 (ablation) | IGM object discovery · CLEVR, 3-5 objects | perception rate: 72.8% -> 90.5% | Helped |
| Use 15 random initializations instead of 1 (ablation) | IGM object discovery · CLEVR, 3-5 objects | perception rate: 72.8% -> 92.8% | Helped |
| Use 20 random initializations instead of 1 (ablation) | IGM object discovery · CLEVR, 3-5 objects | perception rate: 72.8% -> 94.7% | Helped |
| Use 5 random initializations instead of 1 (ablation) | IGM object discovery · CLEVR, 6-8 objects | perception rate: 68.0% -> 79.1% | Helped |
| Use 10 random initializations instead of 1 (ablation) | IGM object discovery · CLEVR, 6-8 objects | perception rate: 68.0% -> 81.6% | Helped |
| Use 15 random initializations instead of 1 (ablation) | IGM object discovery · CLEVR, 6-8 objects | perception rate: 68.0% -> 84.3% | Helped |
| Use 20 random initializations instead of 1 (ablation) | IGM object discovery · CLEVR, 6-8 objects | perception rate: 68.0% -> 85.3% | Helped |
| Use continuous approximation for concept inference (ablation) | IGM global-factor perception · CelebA, 4 attributes, male faces | inference time: 29.10s -> 22.15s | Helped |
Surprising findings
- Naive prompt weighting (Compel) performs much worse than the baseline Diffusion Classifier (35.2% vs 70.4%) on zero-shot multi-object perception.
Limitations
- Enumeration over discrete concept configurations scales exponentially (M^K) with the number of concepts.
- Assumes conditional independence among concepts given the image, ignoring object interactions.
- Evaluated only on simple datasets (CLEVR, CelebA) plus a small 71-image real-world set; CLEVRTex results are limited.
Experimental setup
Datasets: CLEVR CLEVRTex CelebA manually collected 71 internet animal images
Benchmarks: CLEVR CLEVRTex CelebA manually collected animal image set
Metrics: perception rate coordinate estimation error classification accuracy inference time
Baselines: ResNet-50 Slot Attention DINOSAUR Generative Classifier Diffusion Classifier Compel
Builds on: Generative Classifier Diffusion Classifier compositional diffusion models (product of EBMs) Slot Attention classifier-free guidance
Closest papers by meaning
- GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations (ICLR 2020 · similarity 0.91)
- Unsupervised Learning of Compositional Scene Representations from Multiple Unspecified Viewpoints (AAAI 2022 · similarity 0.90)
- Generative Scene Graph Networks (ICLR 2021 · similarity 0.90)
- Generative Modeling of Infinite Occluded Objects for Compositional Scene Representation (ICML 2019 · similarity 0.90)
- The Scene Language: Representing Scenes with Programs, Words, and Embeddings (CVPR 2025 · similarity 0.88)
- Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.