Research map / Image generation / Text-to-image diffusion
Trade-Offs in Fine-Tuned Diffusion Models between Accuracy and Interpretability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that fine-tuning Stable Diffusion v2 on chest X-ray reports with a learnable text encoder destroys phrase-grounding interpretability, and that freezing the text encoder keeps localization via cross-attention maps at some cost in FID.
Problem
Fine-tuned text-to-image diffusion models for medical imaging (e.g., chest X-ray synthesis from radiology reports) may not understand what their text conditioning means, which undermines interpretability and trust in generated content.
Approach
Compare learnable versus frozen text encoders when fine-tuning Stable Diffusion v2 on CXR data and use cross-attention maps over disease-word tokens for zero-shot phrase grounding; show that a frozen encoder preserves interpretability while jointly learning the encoder destroys it, creating a fidelity-interpretability trade-off.
Key results
- On MS-CXR phrase grounding, the frozen-encoder Finetune_F reaches 79% average AUC-ROC, 45% Top-1 and 0.920 CNR, versus 54% AUC for the learnable encoder Finetune_L (CNR 0.133).
- Finetune_F achieves FID 75.5 on MIMIC versus 61.9 for the learnable Finetune_L, showing a fidelity cost for the frozen encoder.
- Finetune_F beats the discriminative BioViL-L on two of eight diseases in CNR-based phrase grounding (e.g., Atelectasis 1.25, Edema 1.36 versus BioViL-L's lower values).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Freeze the language encoder instead of making it learnable during fine-tuning | Stable Diffusion v2 fine-tuned on CXR data · MS-CXR | AUC-ROC: 54±3.7 -> 79±0.5 | Helped |
| Freeze the language encoder instead of making it learnable during fine-tuning | Stable Diffusion v2 fine-tuned on CXR data · MS-CXR | FID_XRV: 7.3 -> 10.0 | Hurt |
| Freeze the language encoder instead of making it learnable during fine-tuning | Stable Diffusion v2 fine-tuned on CXR data · MS-CXR synthetic images | classifier AUC-ROC: 77.3 -> 77.1 | No clear effect |
| Freeze the language encoder during MS-COCO fine-tuning for 150000 steps | Stable Diffusion v2 · MS-COCO | FID: 25.0 -> 19.4 | Helped |
| Disable unconditional guidance training (ablation) | Finetune_F · CXR generation | FID_XRV: 10.0 -> 16.4 | Hurt |
| Disable unconditional guidance training (ablation) | Finetune_F · MS-CXR | AUC-ROC: 79 -> 78.6 | Hurt |
| Continue learnable-encoder fine-tuning for 10000 steps with the encoder frozen (ablation) | Finetune_L · CXR generation | FID_XRV: 7.3 -> 9.3 | Hurt |
| Continue learnable-encoder fine-tuning for 10000 steps with the encoder frozen (ablation) | Finetune_L · MS-CXR | AUC-ROC: 54±3.7 -> 52.1 | No clear effect |
| Train only on PA images (ablation) | Finetune_F · CXR generation | FID_XRV: 10.0 -> 3.6 | Helped |
| Train only on PA images (ablation) | Finetune_F · MS-CXR | AUC-ROC: 79 -> 76.9 | Hurt |
| Use ChatGPT-generated impressions rather than radiology impressions (ablation) | Frozen-encoder fine-tuning · MS-CXR subset with labels in impressions | CNR: 0.95 -> 0.42 | Hurt |
| Use class labels rather than radiology impressions as text input (ablation) | Frozen-encoder fine-tuning · MS-CXR subset with labels in impressions | CNR: 0.95 -> 0.80 | Hurt |
Surprising findings
- Jointly learning the text encoder yields better FID but near-zero or negative localization, meaning high image fidelity does not imply the model understands its conditioning.
- The learnable-encoder model's best-localizing token is often the end-of-string token rather than the disease word, suggesting it learns positional shortcuts.
Limitations
- Localization relies on hand-picked disease keyword tokens, and CNR evaluation uses bounding boxes from only 8 disease classes.
- FID is a limited generative metric for medical images and conditional quality could not be validated by ground-truth or expert review.
- Results are reported over only three to five runs with substantial standard deviations for some classes.
Experimental setup
Datasets: MS-COCO MIMIC-CXR MS-CXR ChatGPT-generated impressions
Benchmarks: MS-CXR MS-COCO MIMIC-CXR p19 subset
Metrics: AUC-ROC Top-1 accuracy CNR FID FID_XRV MS-SSIM
Baselines: SDv2 (pretrained, no fine-tuning) Finetune_L (learnable text encoder) ConVIRT GLoRIA BioViL-L
Builds on: Stable Diffusion v2 classifier-free guidance MS-CXR MIMIC-CXR
Closest papers by meaning
- Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generation (CVPR 2024 · similarity 0.87)
- Are Images Indistinguishable to Humans Also Indistinguishable to Classifiers? (CVPR 2025 · similarity 0.87)
- Latent Drifting in Diffusion Models for Counterfactual Medical Image Synthesis (CVPR 2025 · similarity 0.86)
- Where to Look Is Not How to Fix: Pre-Denoising Diagnostics and Modality-Dependent Control in Diffusion Composition (NeurIPS 2026 · similarity 0.86)
- Precise Parameter Localization for Textual Generation in Diffusion Models (ICLR 2025 · similarity 0.86)
- DifFRACT: Diffusion Feature Reconstruction and Attribution for Circuit Tracing (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.