Research map / Image generation / Autoregressive and latent generation
Generative Medical Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Generative Medical Segmentation (GMS), which maps image latents to mask latents with a lightweight convolutional/self-attention latent mapping model on top of a frozen Stable Diffusion VAE, and decodes back to masks.
Problem
Medical image segmentation with CNNs and ViTs has limited generalization across datasets and high parameter counts that risk overfitting on small medical datasets; prior generative segmentation approaches (diffusion, GSS) are slow or parameter-heavy.
Approach
Freeze a pre-trained VAE (Stable Diffusion VAE) as an image tokenizer and train only a small latent mapping model that predicts mask latents from image latents, with a combined latent-space MSE loss and image-space Dice loss; the frozen decoder turns predicted latents into masks.
Key results
- GMS reaches 88.42 DSC on BUS and 81.43 DSC on BUSI with only 1.5M trainable parameters, outperforming nnUNet (85.71 and 79.45).
- On GlaS, HAM10000 and Kvasir-Instrument GMS reports DSC of 88.98, 94.11 and 94.24, the best in each case except the Kvasir-Instrument IoU, where nnUNet (90.20) is higher.
- In BUSI-to-BUS and BUS-to-BUSI domain generalization, GMS reaches 80.31 and 61.60 DSC, above nnUNet (78.39 and 59.13).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add segmentation loss to latent matching loss (ablation) | GMS · BUSI, HAM10000, and Kvasir | DSC: 80.25 -> 81.43; 93.92 -> 94.11; 92.93 -> 94.24 | Helped |
| add latent matching loss to segmentation loss (ablation) | GMS · BUSI, HAM10000, and Kvasir | DSC: 78.75 -> 81.43; 93.64 -> 94.11; 93.00 -> 94.24 | Helped |
| replace VQ-VAE tokenizer with SD-VAE (ablation) | GMS · BUSI, HAM10000, and Kvasir | DSC: 79.23 -> 81.43; 92.77 -> 94.11; 92.47 -> 94.24 | Helped |
| use GMS instead of nnUNet | medical segmentation · BUS | DSC: 85.71 -> 88.42 | Helped |
| use GMS instead of GSS | medical segmentation · BUSI | DSC: 79.56 -> 81.43 | Helped |
| use GMS instead of ACC-UNet | medical segmentation · GlaS | DSC: 88.60 -> 88.98 | Helped |
| use GMS instead of EGE-UNet | medical segmentation · HAM10000 | DSC: 93.90 -> 94.11 | Helped |
| use GMS instead of nnUNet | medical segmentation · Kvasir-Instrument | DSC: 93.95 -> 94.24 | Helped |
| use GMS instead of nnUNet | cross-center ultrasound segmentation · BUSI to BUS | DSC: 78.39 -> 80.31 | Helped |
| use GMS instead of nnUNet | cross-center ultrasound segmentation · BUS to BUSI | DSC: 59.13 -> 61.60 | Helped |
Surprising findings
- Segmentation performance does not correlate with trainable parameter count; EGE-UNet with fewer parameters underfits while SME-SwinUNet and MedSegDiff-V2 overfit.
- Transformer-based models underperform CNNs on grayscale ultrasound datasets.
Limitations
- Limited to 2D medical images because the SD-VAE tokenizer is trained only on 2D natural images.
- Image size fixed to 224x224 and experiments use small datasets with random 80/20 splits for non-divided datasets.
- Loss-function ablation shows the optimal supervision balance differs across datasets.
Experimental setup
Datasets: BUS BUSI GlaS HAM10000 Kvasir-Instrument
Benchmarks: BUS BUSI GlaS HAM10000 Kvasir-Instrument
Metrics: DSC IoU HD95
Baselines: UNet MultiResUNet ACC-UNet nnUNet EGE-UNet SwinUNet SME-SwinUNet UCTransNet MedSegDiff-V2 SDSeg GSS MixStyle
Builds on: Stable Diffusion VAE GSS MedSegDiff-V2 nnU-Net AdamW
Closest papers by meaning
- Why Invariance is Not Enough for Biomedical Domain Generalization and How to Fix It (NeurIPS 2026 · similarity 0.87)
- Learning Generalized Medical Image Segmentation from Decoupled Feature Queries (AAAI 2024 · similarity 0.87)
- Tyche: Stochastic In-Context Learning for Medical Image Segmentation (CVPR 2024 · similarity 0.86)
- LangDAug: Langevin Data Augmentation for Multi-Source Domain Generalization in Medical Image Segmentation (ICML 2025 · similarity 0.86)
- Interactive Medical Image Segmentation: A Benchmark Dataset and Baseline (CVPR 2025 · similarity 0.86)
- Enhancing SAM with Efficient Prompting and Preference Optimization for Semi-supervised Medical Image Segmentation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.