Research map / Image generation / Text-to-image diffusion
Rethinking FID: Towards a Better Evaluation Metric for Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that FID is unreliable for modern text-to-image evaluation and proposes CMMD, an MMD-based metric with CLIP ViT-L/14@336px embeddings and a Gaussian RBF kernel.
Problem
FID, the de facto metric for image generation quality, relies on Inception-v3 features and a multivariate normality assumption, and it disagrees with human raters, fails to track iterative improvement or distortions, and is sample-inefficient.
Approach
Replace Inception features and the Fréchet distance with CLIP embeddings and an unbiased squared MMD estimate using a Gaussian RBF kernel, which is distribution-free, unbiased, and sample-efficient.
Key results
- Human raters preferred Muse full model over early-stopped model 92.5% vs 6.9%, while FID (21.40 vs 18.42) favored the worse model; CMMD (0.721 vs 0.951) agreed with humans.
- Under VQGAN-latent token distortions, FID improved for the first distortion levels while CMMD increased monotonically with distortion level.
- FID required more than 20,000 images for reliable estimates, whereas CMMD gave consistent estimates with small sets; MMD took 71 ms versus 7008 ms for Fréchet distance on 30K samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate early-stopped Muse (20 base-model iterations and 3 super-resolution iterations) instead of full Muse (24 base-model iterations and 8 super-resolution iterations) | FID · PartiPrompts | FID: 21.40 -> 18.42 | Hurt |
| Evaluate early-stopped Muse instead of full Muse | FID∞ · PartiPrompts | FID∞: 20.16 -> 17.19 | Hurt |
| Evaluate early-stopped Muse instead of full Muse | KID · PartiPrompts | KID: 0.0105 -> 0.0080 | Hurt |
| Evaluate early-stopped Muse instead of full Muse | CMMD · PartiPrompts | CMMD: 0.721 -> 0.951 | Helped |
| Use CMMD instead of FID to assess progressive Muse refinement | Muse · Muse refining iterations | alignment with image-quality improvement | Helped |
| Use CMMD instead of FID to assess Stable Diffusion refinement | Stable Diffusion · Stable Diffusion steps 95–100 | alignment with image-quality improvement | Helped |
| Apply VQGAN token replacement with probability p = 0.2 | Muse images evaluated with FID · VQGAN latent-space distortion | FID: 21.40 -> 18.02 | Helped |
| Apply VQGAN token replacement with probability p = 0.2 | Muse images evaluated with CMMD · VQGAN latent-space distortion | CMMD: 0.721 -> 1.190 | Hurt |
| Use CMMD rather than FID for evaluation across sample sizes | Stable Diffusion · COCO 30K sample-size evaluation | consistency of metric estimates | Helped |
Surprising findings
- FID and FID∞ suggested Muse image quality degrades over the final refinement iterations, contrary to visual evidence.
- FID improved for the first few levels of latent-space distortion, indicating better quality under visibly degraded images.
Limitations
- Human evaluation covers only one model pair (Muse full vs early-stopped) on PartiPrompts.
- CMMD bandwidth fixed at sigma=10 and features scaled by 1000 by convention; sensitivity is only described qualitatively.
- Relies on COCO 30K as reference set, a standard that the paper itself notes is de facto but limited.
Experimental setup
Datasets: COCO 30K PartiPrompts WebLI
Benchmarks: COCO 30K PartiPrompts
Metrics: FID FID∞ KID CMMD human preference rate Cohen's kappa
Baselines: FID FID∞ KID
Builds on: FID KID MMD CLIP
Closest papers by meaning
- Representation Fréchet Loss for Visual Generation (NeurIPS 2026 · similarity 0.90)
- Fréchet Wavelet Distance: A Domain-Agnostic Metric for Image Generation (ICLR 2025 · similarity 0.87)
- STREAM: Spatio-TempoRal Evaluation and Analysis Metric for Video Generative Models (ICLR 2024 · similarity 0.86)
- MIND: Monge Inception Distance for Generative Models Evaluation (NeurIPS 2026 · similarity 0.86)
- FoMo: Forking Moment in Generative Trajectory as a Perceptual Distance (NeurIPS 2026 · similarity 0.85)
- On the Content Bias in Frechet Video Distance (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.