Research map / Image generation / Autoregressive and latent generation
Explore In-Context Segmentation via Latent Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Turns Stable Diffusion 1.5 into an in-context segmenter (LDIS) that takes visual prompts (via Alpha-CLIP prompt encoder with two-stage masking) and outputs masks via one-step or multi-step latent diffusion, trained on PASCAL, COCO, DAVIS-16 and VSPW.
Problem
In-context segmentation: segmenting query-image objects specified only by reference image-mask prompts, without text prompts or auxiliary decoders, using latent diffusion models.
Approach
Reframes in-context segmentation as conditional mask generation with a minimal SD backbone, conditioned only on visual instructions extracted by a prompt encoder with two-stage (pre- and post-) masking, and trained with pseudo-mask targets augmented by the query image; proposes LDIS-1 (one-step, pixel-space loss) and LDIS-n (multi-step noise prediction).
Key results
- LDIS-1 reaches 85.3 mIoU on PASCAL and 62.6 mIoU on COCO, outperforming SegGPT (75.6 and 50.7) in the reported table.
- LDIS-n reaches 76.7 mIoU on PASCAL and 52.6 mIoU on COCO; on video, LDIS-1 scores 67.8 J&F on DAVIS-16 and 41.6 mIoU on VSPW, below SegGPT.
- Two-stage masking raises LDIS-n COCO mIoU from 43.8 (pre-mask only) and 49.7 (post-mask only) to 52.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LDIS-1 instead of the strongest listed baseline | PASCAL · PASCAL | mIoU: 77.0 -> 85.3 | Helped |
| Use LDIS-1 instead of the strongest listed baseline | COCO · COCO | mIoU: 52.1 -> 62.6 | Helped |
| Use LDIS-1 instead of SegGPT | video segmentation · DAVIS-16 | J&F: 80.5 -> 67.8 | Hurt |
| Use LDIS-1 instead of SegGPT | video semantic segmentation · VSPW | mIoU: 61.7 -> 41.6 | Hurt |
| Increase visual prompts from 1 to 10 (ablation) | LDIS-n · COCO | mIoU: 49.7 -> 56.7 | Helped |
| Increase visual prompts from 10 to 20 (ablation) | LDIS-n · COCO | mIoU: 56.7 -> 55.7 | Hurt |
| Add pre-masking to post-masking (ablation) | LDIS-n instruction extraction · COCO | mIoU: 49.7 -> 52.6 | Helped |
| Add post-masking to pre-masking (ablation) | LDIS-n instruction extraction · COCO | mIoU: 43.8 -> 52.6 | Helped |
| Add uniform perturbation to vanilla pseudo-masks (ablation) | LDIS-n output alignment · COCO | mIoU: 39.4 -> 48.7 | Helped |
| Fuse the query image into the pseudo-mask target (ablation) | LDIS-n output alignment · COCO | mIoU: 48.7 -> 49.7 | Helped |
| Use pixel-space instead of latent-space optimization (ablation) | LDIS-1 with LoRA rank 4 · COCO | mIoU: 53.2 -> 55.6 | Helped |
| Train all parameters instead of LoRA rank 4 (ablation) | LDIS-1 · COCO | mIoU: 53.2 -> 61.3 | Helped |
Surprising findings
- Perturbing vanilla pseudo masks with uniform noise raises LDIS-n from 39.4 to 48.7 mIoU, suggesting a constant-valued target enables shortcut learning.
- LoRA fine-tuning drops performance by 8.1 mIoU on LDIS-1 and 26.5 on LDIS-n, indicating generation-pretrained expressive power is needed for segmentation.
Limitations
- Does not achieve state-of-the-art on all video metrics; performance trails SegGPT on VSPW and comparable-only results on DAVIS-16.
- Uses no spatial prompts, so it lacks spatial priors that generalists exploit for video.
- Training data is much smaller than foundation-model generalists, which the authors note explains part of the gap.
Experimental setup
Datasets: PASCAL COCO DAVIS-16 VSPW
Benchmarks: PASCAL COCO DAVIS-16 VSPW COCO-20i
Metrics: mIoU FB-IoU J&F fwIoU
Baselines: PFENet SVF VTM DCAMA VPD Painter SegGPT PerSAM Prompt Diffusion
Builds on: Stable Diffusion classifier-free guidance Alpha-CLIP LoRA
Closest papers by meaning
- Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation (NeurIPS 2024 · similarity 0.90)
- VidSeg: Training-free Video Semantic Segmentation based on Diffusion Models (CVPR 2025 · similarity 0.89)
- Prompting Diffusion Models for Zero-Shot Instance Segmentation (NeurIPS 2026 · similarity 0.88)
- A Simple Image Segmentation Framework via In-Context Examples (NeurIPS 2024 · similarity 0.88)
- $\mathbf{D^{3}S^{2}}$: Diffusion-Guided Dataset Distillation for Semantic Segmentation (NeurIPS 2026 · similarity 0.88)
- SeeDiff: Off-the-Shelf Seeded Mask Generation from Diffusion Models (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.