Research map / Image generation / Text-to-image diffusion
Explaining in Diffusion: Explaining a Classifier with Diffusion Semantics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiffEx, a training-free method that uses a VLM-built hierarchical semantic corpus and diffusion-based editing (Ledits++ on SDXL) with a beam-search-like procedure to rank semantics that influence classifier logits.
Problem
Explaining the decisions of black-box classifiers by identifying fine-grained, hierarchical semantic attributes that influence their outputs; prior GAN-based methods (StylEx) are limited to single concepts like faces and need a new GAN per classifier.
Approach
Replace GAN latent editing with text-to-image diffusion editing (Ledits++) driven by a VLM-extracted hierarchical attribute corpus, and search that hierarchy beam-search style, keeping child semantics only when they raise the classifier's logit influence over their parent.
Key results
- In a user study on the bird domain, DiffEx achieved 88.4% average correct attribute selection versus 76.5% for StylEx and 53% for Grad-CAM.
- User study 1 rated edit quality 3.386 (bird) and 3.659 (face) on a 1-5 scale with disentanglement 3.163 and 3.204.
- Ablation: VLM semantic consistency IoU vs GPT-4 was 0.94 (Llama 3.2), 0.87 (Qwen2-VL), 0.93 (Claude 3.5 Haiku); diffusion attribute nDCG 0.75 (SD 1.5), 0.76 (SD 2.0) against SDXL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DiffEx instead of Grad-CAM | bird-class attribute explanation · Bird class user study | average correct responses: 53% -> 88.4% | Helped |
| use DiffEx instead of StylEx | bird-class attribute explanation · Bird class user study | average correct responses: 76.5% -> 88.4% | Helped |
| use Llama 3.2 as the VLM (ablation) | DiffEx · Face domain | semantic consistency IoU against GPT-4: 0.94 | No clear effect |
| use Qwen2-VL as the VLM (ablation) | DiffEx · Face domain | semantic consistency IoU against GPT-4: 0.87 | No clear effect |
| use Claude 3.5 Haiku as the VLM (ablation) | DiffEx · Face domain | semantic consistency IoU against GPT-4: 0.93 | No clear effect |
| use Stable Diffusion 1.5 instead of SDXL (ablation) | DiffEx · Semantic attribute ranking | nDCG against SDXL attributes: 0.75 | No clear effect |
| use Stable Diffusion 2.0 instead of SDXL (ablation) | DiffEx · Semantic attribute ranking | nDCG against SDXL attributes: 0.76 | No clear effect |
Surprising findings
- Explanations reveal dataset biases, e.g., a collar strongly increases the likelihood of a dog classification, likely due to training data correlations.
Limitations
- Quality and scope of the VLM-curated semantic corpus limit which features can be found and may miss relevant ones.
- Off-the-shelf Ledits++ editing can produce entangled edits and fails on fine details such as earlobes or nose rings.
Experimental setup
Datasets: CelebA-style facial attribute data (age/gender) plant health dataset retinal fundus dataset bird species dataset (CUB-style) wildcat/pet species dataset fashion dataset places dataset food dataset ISIC NIH ChestX-ray14 CheXpert
Benchmarks: Face (age) classifier bird species classifier wildcat/pet classifier retinal disease classifier (FLAIR) plant health classifier
Metrics: average logit influence score user-study edit quality and disentanglement ratings (1-5) percentage of correct attribute selections IoU vs GPT-4 for VLM semantic consistency nDCG for diffusion model attribute overlap
Baselines: StylEx Grad-CAM FastDiME FastDiME-2+ DiME ACE
Builds on: StylEx Ledits++ FastDiME beam search
Closest papers by meaning
- Diffusion Classifiers Understand Compositionality, but Conditions Apply (NeurIPS 2025 · similarity 0.87)
- NoiseCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions in Diffusion Models (CVPR 2024 · similarity 0.86)
- Language Model as Visual Explainer (NeurIPS 2024 · similarity 0.85)
- Comparing the Decision-Making Mechanisms by Transformers and CNNs via Explanation Methods (CVPR 2024 · similarity 0.85)
- The Hidden Language of Diffusion Models (ICLR 2024 · similarity 0.85)
- I-CEE: Tailoring Explanations of Image Classification Models to User Expertise (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.