Research map / Image generation / Text-to-image diffusion
Diffusion Classifiers Understand Compositionality, but Conditions Apply
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Comprehensive study of zero-shot diffusion classifiers (SD1.5, SD2.0, SD3-m) across 10 compositional benchmarks and 33 tasks, introducing the Self-Bench diagnostic set and low-shot timestep reweighting to address domain gaps.
Problem
Diffusion classifiers have shown promising but preliminary compositional discrimination results on few benchmarks; it is unclear when and why they succeed compared to CLIP, and how domain gaps and timestep choice affect them.
Approach
Evaluates diffusion classifiers broadly on compositional benchmarks, introduces Self-Bench (images generated by the diffusion models themselves) to isolate domain effects, and learns low-shot task- and model-specific timestep weightings that mitigate domain gaps, especially for SD3-m, which is the first diffusion classifier study with SD3-m.
Key results
- Diffusion classifiers excel on spatial position tasks but underperform CLIP on object recognition and counting; SD3-m reaches 39% average discriminative accuracy versus 43% for earlier SD versions.
- On Self-Bench, in-domain discrimination correlates with generation accuracy (correlation coefficient 0.77), while SD3-m shows the largest cross-domain accuracy drop.
- Low-shot timestep reweighting with 5% of target data substantially improves SD3-m, e.g., 98% on CLEVR binding (vs. 63% baseline) and 42% on WhatsUp spatial (vs. 30%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SD3-m instead of earlier Stable Diffusion versions | Diffusion Classifier · Compositional benchmarks | accuracy: 43% -> 39% | Hurt |
| apply low-shot timestep reweighting | SD3-m · CLEVR binding | accuracy: 63% -> 98% | Helped |
| apply low-shot timestep reweighting | SD3-m · WhatsupA spatial task | accuracy: 30% -> 42% | Helped |
| evaluate on a different diffusion model's generated images rather than the model's own generated images | Diffusion Classifier · Self-Bench | accuracy | Hurt |
| use higher-resolution no-resize inputs (ablation) | SD3-m · Geneval counting | accuracy: 58.26% -> 96.30% | Helped |
| use higher-resolution no-resize inputs (ablation) | SD3-m · Geneval color attribution | accuracy: 97.22% -> 98.09% | Helped |
| increase sampled timesteps from 30 to 100 (ablation) | Diffusion Classifiers · Geneval tasks | accuracy: 83.33% -> 66.67%; 73.68% -> 84.21% | Mixed |
| use logit-normal timestep weighting instead of uniform weighting (ablation) | SD3-m · Self-Bench cross-domain | accuracy: did not exceed 11% | Hurt |
| use exponential timestep weighting exp(-7t) instead of uniform weighting (ablation) | SD3-m · CLEVR Binding–Color | accuracy: 0.60 -> 0.75 | Helped |
| use exponential timestep weighting exp(-7t) instead of uniform weighting (ablation) | SD3-m · Spec Count | accuracy: 0.19 -> 0.11 | Hurt |
| sample classification timesteps from t ∼ [0.5, 1] (ablation) | SD3-m · Self-Bench2.0 Single | accuracy: 0.87 -> 0.90 | Helped |
| sample classification timesteps from t ∼ [0.5, 1] (ablation) | SD3-m · CLEVR Binding | accuracy: 0.63 -> 0.59 | Hurt |
Surprising findings
- SD3-m, despite stronger generative compositionality, achieves lower discriminative accuracy than earlier SD1.5 and SD2.0 models on real compositional benchmarks.
- Uniform timestep weighting performs better for SD3 classification than its native logit-normal weighting, which yielded under 11% cross-domain accuracy.
Limitations
- Timestep weighting gains are limited to SD3-m; SD1.5 and SD2.0 do not benefit significantly.
- No error bars or statistical significance reported.
- Domain gap is approximated with CLIP embedding distances, which conflate stylistic and semantic shifts.
Experimental setup
Datasets: Vismin EQBench MMVP CLEVR Whatsup Spec ARO Sugarcrepe COLA Winoground GenEval
Benchmarks: Vismin EQBench MMVP CLEVR Whatsup Spec ARO Sugarcrepe COLA Winoground Self-Bench
Metrics: image-to-text retrieval accuracy generation accuracy
Baselines: CLIP RN50x64 CLIP ViT-B/32 CLIP ViT-L/14 CLIP ViT-H/14 CLIP ViT-g/14 SigLIP SigLIP2 Discffusion SD1.5 SD2.0
Builds on: Diffusion Classifier (Li et al.) Clark et al. timestep weighting Discffusion Stable Diffusion 3 GenEval
Closest papers by meaning
- Not All Diffusion Model Activations Have Been Evaluated as Discriminative Features (NeurIPS 2024 · similarity 0.90)
- Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Models (NeurIPS 2025 · similarity 0.89)
- How Diffusion Models Learn to Factorize and Compose (NeurIPS 2024 · similarity 0.89)
- ORCA: Hunting Compositional Failures in Text-to-Image Diffusion (NeurIPS 2026 · similarity 0.89)
- Mining your own secrets: Diffusion Classifier Scores for Continual Personalization of Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.88)
- How Compositional Generalization and Creativity Improve as Diffusion Models are Trained (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.