Research map / Image generation / Fast diffusion sampling
Latent Drifting in Diffusion Models for Counterfactual Medical Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Latent Drift, a latent-space offset added to diffusion forward/reverse processes that can be combined with any fine-tuning method to adapt Stable Diffusion to medical counterfactual image generation and manipulation on brain MRI and chest X-rays.
Problem
Fine-tuning general pre-trained diffusion models for medical imaging suffers from distribution shift and scarce data, making counterfactual medical image generation (aging, disease, gender changes) hard.
Approach
Introduce a signed scalar latent drift δ added to the mean of the reverse diffusion kernel (and the forward target latent during fine-tuning), tuned via grid search on an L1 distance between generated and target samples, framed as a counterfactual min-max objective; it is agnostic to the fine-tuning method.
Key results
- On brain MRI, Stable Diffusion basic fine-tuning FID dropped from 92.13 to 49.68 with latent drift.
- On CheXpert, Textual Inversion FID improved from 171.77 to 133.18 with latent drift, and Stable Diffusion basic FT FID from 112 to 84.
- Classifiers trained on Real+Synthetic data (50% split) reached AUC 0.883 on brain MRI versus 0.870 for real data alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Latent Drifting to Stable Diffusion basic fine-tuning | Stable Diffusion v1.4 basic fine-tuning · Brain MR | FID: 92.13 -> 49.68 | Helped |
| Add Latent Drifting to Stable Diffusion basic fine-tuning | Stable Diffusion v1.4 basic fine-tuning · Brain MR | AUC: 0.704 -> 0.724 | Helped |
| Add Latent Drifting to Stable Diffusion basic fine-tuning | Stable Diffusion v1.4 basic fine-tuning · CheXpert | FID: 112 -> 84 | Helped |
| Add Latent Drifting to Stable Diffusion basic fine-tuning | Stable Diffusion v1.4 basic fine-tuning · CheXpert | AUC: 0.672 -> 0.746 | Helped |
| Use diverse prompts with patient information instead of simple prompts without patient information (ablation) | Stable Diffusion + Basic FT + LD · Text-to-image generation | FID: 60.41 -> 51.35 | Helped |
| Use diverse prompts with patient information instead of simple prompts without patient information (ablation) | Pix2Pix Zero + LD · Image-to-image generation | FID: 46.79 -> 36.10 | Helped |
| Use numerical age prompts instead of binned age prompts (ablation) | InstructPix2Pix + LD · Age-conditioned manipulation | FID: 17.05 -> 16.27 | Helped |
| Use numerical age prompts instead of word-based age prompts (ablation) | InstructPix2Pix + LD · Age-conditioned manipulation | FID: 15.84 -> 16.27 | No clear effect |
Surprising findings
- Custom Diffusion performed much worse than other backbones for age-conditioned manipulation despite realistic-looking samples in text-to-image generation.
Limitations
- Latent drift value δ is selected by grid search, requiring tuning per setting.
- Evaluated on only three public datasets with small sample sizes (800 chest X-rays, 3269 brain scans).
Experimental setup
Datasets: ADNI-1 OASIS-3 CheXpert LAION-5B
Benchmarks: ADNI-1 OASIS-3 CheXpert
Metrics: FID KID AUC SSIM LPIPS PSNR
Baselines: Textual Inversion DreamBooth Custom Diffusion Stable Diffusion basic fine-tuning InstructPix2Pix Pix2Pix Zero
Builds on: Stable Diffusion DreamBooth Textual Inversion Custom Diffusion InstructPix2Pix Pix2Pix Zero
Closest papers by meaning
- Trade-Offs in Fine-Tuned Diffusion Models between Accuracy and Interpretability (AAAI 2024 · similarity 0.86)
- ZoomLDM: Latent Diffusion Model for Multi-scale Image Generation (CVPR 2025 · similarity 0.86)
- Alias-Free Latent Diffusion Models: Improving Fractional Shift Equivariance of Diffusion Latent Space (CVPR 2025 · similarity 0.86)
- Autoregressive Image Diffusion: Generation of Image Sequence and Application in MRI (NeurIPS 2024 · similarity 0.85)
- Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models (CVPR 2024 · similarity 0.85)
- Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.