Research map / Image generation / Fast diffusion sampling
Semantic and Visual Crop-Guided Diffusion Models for Heterogeneous Tissue Synthesis in Histopathology
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HeteroTissue-Diffuse, a latent diffusion model conditioned on semantic maps plus raw tissue crops for heterogeneous histopathology synthesis, and extends it to unannotated TCGA via foundation-model clustering into 100 pseudo-tissue types.
Problem
Synthetic histopathology generation struggles to preserve tissue heterogeneity, fine morphology, and scale to unannotated WSI collections; existing methods rely on text prompts, class labels, or abstract embeddings.
Approach
Condition a latent diffusion model on concatenated per-class binary semantic masks and raw RGB tissue crops placed inside the corresponding masked regions, and generate pseudo-semantic maps for TCGA by clustering UNI patch embeddings into 100 tissue types.
Key results
- On Camelyon16, visual-prompt FD with RN50-BT drops from 430.1 (non-prompt) to 72.0.
- DeepLabv3+ trained on synthetic data reaches test IoU 0.71 (Camelyon16) and 0.95 (PANDA) vs 0.72 and 0.96 with real data.
- A certified pathologist classified real vs synthetic images at roughly chance level (45% accuracy on Camelyon16).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use visual prompts instead of no prompting | HeteroTissue-Diffuse on Camelyon16 · Camelyon16 | FD (RN50-BT): 430.1 -> 72.0 | Helped |
| Use visual prompts instead of no prompting | HeteroTissue-Diffuse on PANDA · PANDA | FD (RN50-BT): 150.0 -> 22.8 | Helped |
| Use visual prompts instead of no prompting | HeteroTissue-Diffuse on TCGA · TCGA | FD (RN50-BT): 157.7 -> 142.9 | Helped |
| Train DeepLabv3+ on prompt-conditioned synthetic data instead of NoPrompt synthetic data | DeepLabv3+ on Camelyon16 · Camelyon16 test | IoU: 0.63 -> 0.71 | Helped |
| Train DeepLabv3+ on prompt-conditioned synthetic data instead of NoPrompt synthetic data | DeepLabv3+ on PANDA · PANDA test | IoU: 0.86 -> 0.95 | Helped |
| Add visual crops to semantic-map conditioning (ablation) | HeteroTissue-Diffuse on TCGA · TCGA | FID (GigaPath): 360.4 to 346.1 | Helped |
| Add visual crops to semantic-map conditioning (ablation) | HeteroTissue-Diffuse on PANDA · PANDA | FID (RN50-BT): 150.0 to 22.8 | Helped |
| Replace crop embeddings with direct visual crops (ablation) | HeteroTissue-Diffuse on Camelyon16 · Camelyon16 | FD (RN50-BT): 183.0 -> 72.0 | Helped |
| Add visual prompts rather than use no prompt conditioning (ablation) | HeteroTissue-Diffuse on Camelyon16 · Camelyon16 | FD (DINOv2): 122.0 -> 52.7 | Helped |
| Add visual crops to semantic-map conditioning (ablation) | HeteroTissue-Diffuse on TCGA · TCGA | Precision (GigaPath): 0.754 to 0.840 | Helped |
| Add visual crops to semantic-map conditioning (ablation) | HeteroTissue-Diffuse on TCGA · TCGA | Recall (GigaPath): 0.018 to 0.008 | Hurt |
| Add visual crops to semantic-map conditioning (ablation) | HeteroTissue-Diffuse on PANDA · PANDA | F1-Score (GigaPath): 0.007 to 0.438 | Helped |
Surprising findings
- The tissue classifier reaches only 47% top-1 accuracy on TCGA clusters, which the authors attribute to clustering granularity.
- Several FID values in the SM-only vs SM+Crops tables are inconsistent with the text claims (e.g., the TCGA visual-prompt FD for UNI and PathDino is higher or near baseline).
Limitations
- Requires substantial compute for gigapixel WSI processing (feature extraction took about 3 months on one A100).
- Limited to H&E staining; other stains or modalities need adaptation.
- Predefined clustering may miss rare pathological patterns.
Experimental setup
Datasets: Camelyon16 PANDA TCGA
Benchmarks: Camelyon16 PANDA TCGA
Metrics: Fréchet Distance FID precision recall F1-score IoU pathologist Likert ratings
Baselines: NASDM URCDM DiffInfinite Embedding prompt conditioning Non-prompt (semantic map only) conditioning
Builds on: Latent Diffusion Models NASDM URCDM DiffInfinite
Closest papers by meaning
- Learned Representation-Guided Diffusion Models for Large-Image Generation (CVPR 2024 · similarity 0.89)
- Diffusion Generative Modeling for Spatially Resolved Gene Expression Inference from Histology Images (ICLR 2025 · similarity 0.88)
- ZoomLDM: Latent Diffusion Model for Multi-scale Image Generation (CVPR 2025 · similarity 0.88)
- Beyond ViT Tokens: Masked-Diffusion Pretrained Convolutional Pathology Foundation Model for Cell-Level Dense Prediction (NeurIPS 2026 · similarity 0.88)
- STREAM: Stochastic Riemannian Flow Matching with Anisotropic Decoder for Digital Histopathology Image Generation (NeurIPS 2026 · similarity 0.88)
- Dynamic Entity-Masked Graph Diffusion Model for Histopathology Image Representation Learning (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.