Research map / Image generation / Autoregressive and latent generation
Attribute Based Interpretable Evaluation Metrics for Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SaD and PaD, interpretable metrics that compare attribute-strength distributions (single and paired attributes) of generated vs. training images, using a new Heterogeneous CLIPScore for attribute strength.
Problem
Existing generative model metrics (FID, precision, recall, density, coverage) give scores without interpretability about which attributes are over- or under-generated, and cannot capture attribute relationships.
Approach
Measure attribute strength per image via Heterogeneous CLIPScore (cosine similarity between image and text vectors offset from separate image and text centers), then compare per-attribute (SaD) and per-pair (PaD) KDE densities between training and generated sets with KL divergence.
Key results
- HCS achieves 0.817 accuracy and 0.616 F1 on refined CelebA attributes versus 0.798 and 0.575 for CLIPScore.
- SaD and PaD rank ProjectedGAN worst (SaD 17.61e-7, PaD 41.53e-7) despite competitive FID (5.45), and StyleGAN2 best on SaD (7.52e-7).
- SDv1.5 has roughly half the SaD and PaD of SDv2.1 on COCO captions (SaD 24.37 vs 48.23 at N=30) despite worse FID.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use heterogeneous image and text centers instead of standard CLIPScore | CelebA attribute classification · CelebA | accuracy: 0.798 -> 0.817 | Helped |
| Use heterogeneous image and text centers instead of standard CLIPScore | CelebA attribute classification · CelebA | F1 score: 0.575 -> 0.616 | Helped |
| Increase LDM sampling steps from 50 to 200 (ablation) | LDM · FFHQ | SaD ($10^{-7}$): 10.42 -> 14.04 | Hurt |
| Increase LDM sampling steps from 50 to 200 (ablation) | LDM · FFHQ | PaD ($10^{-7}$): 25.36 -> 30.71 | Hurt |
| Use Stable Diffusion v1.5 instead of v2.1 | Stable Diffusion · COCO | SaD ($10^{-7}$): 48.23 -> 24.37 | Helped |
| Use Stable Diffusion v1.5 instead of v2.1 | Stable Diffusion · COCO | PaD ($10^{-7}$): 106.86 -> 60.71 | Helped |
| Use separate image and text embedding centers rather than the entire center (ablation) | Heterogeneous CLIPScore · CelebA refined attributes | accuracy | Helped |
| Replace Heterogeneous CLIPScore with CLIPScore when computing SaD and PaD (ablation) | Attribute divergence metrics · Biased image injection experiment | SaD and PaD response to injected biased images | No clear effect |
| Inject images with a biased attribute distribution (ablation) | FFHQ generated-image set · FFHQ biased data injection | SaD and PaD | Hurt |
| Inject unbiased images matching the training distribution (ablation) | FFHQ generated-image set · FFHQ unbiased data injection | SaD and PaD | No clear effect |
| Corrupt the gender–eyeglasses correlation while keeping marginal attribute distributions identical (ablation) | CelebA set C versus set A · CelebA attribute-pair discernment | PaD: 6.14 -> 9.60 | Hurt |
| Replace images with eyeglasses by images without eyeglasses (ablation) | CelebA set B · CelebA attribute-skip experiment | SaD and PaD: 0.63 -> 0.89; 3.42 -> 4.05 | Hurt |
Surprising findings
- ProjectedGAN has competitive FID and precision/recall but performs poorly on SaD and especially PaD, e.g., generating babies with beards.
- Stable Diffusion v2.1 has better FID than v1.5 but worse SaD and PaD, and LDM with 200 steps scores worse on SaD than 50 steps despite better FID.
Limitations
- KDE requires a large number of samples (over 50K) for stable estimates.
- Metric quality depends on the VLM/BLIP attribute extraction and the chosen attributes.
- Human alignment evaluated only on small CelebA-based surveys with 40 participants.
Experimental setup
Datasets: FFHQ CelebA LSUN Cat COCO LAION-2B
Benchmarks: FFHQ CelebA LSUN Cat COCO LAION-2B
Metrics: SaD PaD FID FID_CLIP Precision Recall Density Coverage accuracy F1 score
Baselines: FID FID_CLIP Precision Recall Density Coverage CLIPScore
Builds on: CLIPScore FID TIFA DiffuseIT
Closest papers by meaning
- RA-ClipScore: Making Generative Model Evaluation More Interpretable (NeurIPS 2026 · similarity 0.89)
- Image Generation Diversity Issues and How to Tame Them (CVPR 2025 · similarity 0.85)
- A Unifying Information-theoretic Perspective on Evaluating Generative Models (AAAI 2025 · similarity 0.85)
- DiverGen: Improving Instance Segmentation by Learning Wider Data Distribution with More Diverse Generative Data (CVPR 2024 · similarity 0.85)
- Balancing Act: Distribution-Guided Debiasing in Diffusion Models (CVPR 2024 · similarity 0.85)
- Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.