Research map / Image generation / Text-to-image diffusion
ICE: Intrinsic Concept Extraction from a Single Image via Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ICE, a training-free-localization, two-stage framework that uses Stable Diffusion v2.1 to automatically extract object-level concepts with masks from a single image and then decompose them into intrinsic (colour, material, shape) concepts via token learning with triplet losses.
Problem
Learning interpretable visual concepts from a single unlabelled image with diffusion-based T2I models is ambiguous; existing methods capture mainly object-level concepts or need human masks/captions or multiple images.
Approach
Uses the T2I model itself for automatic concept localization (CLIP-based SpLiCE text retrieval plus DiffSeg self-attention segmentation, iteratively masking out objects), then a two-phase triplet-loss token learning that decomposes object-level concepts into intrinsic attributes, followed by U-Net and text-encoder fine-tuning.
Key results
- ICE achieves SIMI 0.738 and SIMC 0.822 with CLIP encoder versus ConceptExpress 0.689 and 0.784.
- With DINO encoder ICE reaches SIMI 0.677 versus ConceptExpress 0.319 and ACC1 0.476 versus 0.324.
- Stage One masks reach mIoU 0.635 and recall 0.893 versus ConceptExpress 0.483 and 0.676 on D1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Stage One retrieved masks, Stage Two token initialization, and structured learning (ablation) | ICE · UCE D1, CLIP encoder | SIMI: 0.710 -> 0.738 | Helped |
| Add Stage One retrieved masks, Stage Two token initialization, and structured learning (ablation) | ICE · UCE D1, DINO encoder | SIMI: 0.493 -> 0.677 | Helped |
| Add Stage Two structured concept learning (ablation) | ICE · UCE D1, CLIP encoder | SIMI: 0.726 -> 0.738 | Helped |
| Add Stage Two structured concept learning (ablation) | ICE · UCE D1, DINO encoder | SIMI: 0.501 -> 0.677 | Helped |
| Initialize Stage Two token embeddings with Stage One retrieved text (ablation) | ICE · UCE D1, CLIP encoder | SIMI: 0.722 -> 0.738 | Helped |
| Initialize Stage Two token embeddings with Stage One retrieved text (ablation) | ICE · UCE D1, DINO encoder | SIMI: 0.548 -> 0.677 | Helped |
| Use ICE rather than ConceptExpress for mask generation | D1 mask segmentation · D1 | mIoU: 0.483 -> 0.635 | Helped |
Surprising findings
- Break-A-Scene's SIMI with DINO (0.254) is far below ConceptExpress and ICE, suggesting encoder choice strongly affects reported scores.
Limitations
- Relies on a limited number of optimization steps and concept refinement that fine-tunes the U-Net and text encoder, with intrinsic tokens alone being hard to optimize.
- Evaluation only on a single benchmark (D1) against one main baseline (ConceptExpress).
Experimental setup
Datasets: D1 (Unsplash-based unlabelled images with manually annotated masks)
Benchmarks: Unsupervised Concept Extraction (UCE) benchmark on D1
Metrics: SIMI SIMC ACC1 ACC3 mIoU Recall Precision
Baselines: ConceptExpress Break-A-Scene
Builds on: ConceptExpress Stable Diffusion DiffSeg SpLiCE DreamBooth
Closest papers by meaning
- ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.89)
- The Hidden Language of Diffusion Models (ICLR 2024 · similarity 0.88)
- Learning Vision and Language Concepts for Controllable Image Generation (ICML 2025 · similarity 0.88)
- ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features (ICML 2025 · similarity 0.86)
- Bridging the gap to real-world language-grounded visual concept learning (NeurIPS 2025 · similarity 0.86)
- FACE: Faithful Automatic Concept Extraction (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.