Research map / Image generation / Fast diffusion sampling

EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models

ICLR 2024 · Image generation › Fast diffusion sampling · approach: Diffusion models as perception tools · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds an unsupervised, fine-grained image segmentor from Stable Diffusion v1.4 alone by perturbing low-resolution cross-attention query features and measuring pixel-level changes in generated images, then plugs the masks into annotation-free open-vocabulary models.

Problem

Pre-trained diffusion models' semantic features live in low-resolution layers, so k-means segmentation from them is coarse; it is unclear how much pixel-level semantic knowledge diffusion models hold without extra annotations or priors.

Approach

Modulate a sub-region of 16x16 cross-attention features during SD's denoising, run with +/- offsets, and use the RGB difference map as the pixel-to-mask semantic correspondence strength; each pixel takes the mask with the strongest correspondence, yielding image-resolution masks without training.

Key results

• Unsupervised segmentation with a modified evaluation protocol reaches 33.1 mIoU on ADE20K (AD150) and 45.7 on PASCAL-Context PC59, outperforming SD naive upsampling (29.1 / 41.5) and DINO/STEGO baselines.
• Under the traditional Hungarian-matching protocol on COCO-Stuff-27, the method reaches 26.6 mIoU, on par with STEGO (26.8).
• Plugging the masks into MaskCLIP raises ADE20K mIoU from 11.5 to 15.9 and Cityscapes from 20.9 to 26.5 in annotation-free open-vocabulary segmentation.

Measured effects reported in the paper

Surprising findings

• Under the traditional Hungarian-matching evaluation, the method only performs on par with DINO-based STEGO, which the authors attribute to limitations of that protocol rather than the masks themselves.
• Combining with CLIPpy gives only marginal gains, which the authors attribute to over-smoothed pixel embeddings.

Limitations

• Struggles to distinguish extremely small objects such as animal legs and human faces.
• Feature representations mix object semantics with attributes like location and color, causing over-segmentation of sky and ground.
• Only evaluated with Stable Diffusion; generalization to other generative models is left to future work.

Experimental setup

Datasets: COCO-Stuff PASCAL-Context ADE20K Cityscapes

Benchmarks: ADE20K (AD150) PASCAL-Context (PC59, PC459) COCO-Stuff (CS171, CS27) Cityscapes (City19)

Metrics: mIoU

Baselines: IIC PiCIE DINO STEGO DINOSAUR SegDiscover ACSeg CLIP TCL CLIPpy MaskCLIP GroupViT

Builds on: Stable Diffusion DDPM-based inversion MaskCLIP STEGO Prompt-to-prompt attention injection

Closest papers by meaning

• Seg4Diff: Unveiling Open-Vocabulary Semantic Segmentation in Text-to-Image Diffusion Transformers (NeurIPS 2025 · similarity 0.90)
• Diffusion-Driven Two-Stage Active Learning for Low-Budget Semantic Segmentation (NeurIPS 2025 · similarity 0.89)
• VidSeg: Training-free Video Semantic Segmentation based on Diffusion Models (CVPR 2025 · similarity 0.89)
• High-Precision Dichotomous Image Segmentation via Probing Diffusion Capacity (ICLR 2025 · similarity 0.89)
• $\mathbf{D^{3}S^{2}}$: Diffusion-Guided Dataset Distillation for Semantic Segmentation (NeurIPS 2026 · similarity 0.89)
• Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation (NeurIPS 2024 · similarity 0.88)