Research map / Image generation / Text-to-image diffusion
DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DreamMatcher, a tuning-free plug-in that improves subject fidelity in T2I personalization by replacing self-attention values with semantically matched, mask-filtered reference values while keeping the structure path intact.
Problem
Text-embedding-based T2I personalization often fails to reproduce the reference subject's appearance (colors, textures, shapes), while key-value replacement methods disrupt the target structure path and are limited to local editing.
Approach
Reformulates personalization as semantic matching: appearance matching self-attention warps reference values to the target via diffusion-feature correspondences, filters them with a cycle-consistency confidence mask and foreground mask, leaves query-key structure untouched, and adds a semantic matching guidance term.
Key results
- DreamMatcher on DreamBooth raises IDINO from 0.638 to 0.680 and ICLIP from 0.808 to 0.821 with TCLIP roughly unchanged (0.237 to 0.231).
- On CustomDiffusion, DreamMatcher reaches IDINO 0.700 vs 0.643 for ViCo (51.3M-parameter optimized adapter) without extra fine-tuning.
- User study (45 users, 1440 responses) ranks DreamMatcher highest in subject fidelity (0.36) and prompt fidelity (0.35).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DreamMatcher to DreamBooth | DreamBooth · ViCo dataset | IDINO: 0.638 -> 0.680 | Helped |
| Add DreamMatcher to DreamBooth | DreamBooth · ViCo dataset | ICLIP: 0.808 -> 0.821 | Helped |
| Add DreamMatcher to DreamBooth | DreamBooth · ViCo dataset | TCLIP: 0.237 -> 0.231 | Hurt |
| Add DreamMatcher to DreamBooth | DreamBooth · Challenging non-rigid personalization dataset | IDINO: 0.611 -> 0.655 | Helped |
| Add DreamMatcher to DreamBooth | DreamBooth · Challenging non-rigid personalization dataset | ICLIP: 0.803 -> 0.818 | Helped |
| Add DreamMatcher to DreamBooth | DreamBooth · Challenging non-rigid personalization dataset | TCLIP: 0.242 -> 0.239 | Hurt |
| Add semantic matching (ablation) | DreamBooth · ViCo dataset | IDINO: 0.638 -> 0.683 | Helped |
| Add a semantic-consistent mask to semantic matching (ablation) | DreamBooth with semantic matching · ViCo dataset | TCLIP: 0.201 -> 0.232 | Helped |
| Add a semantic-consistent mask to semantic matching (ablation) | DreamBooth with semantic matching · ViCo dataset | IDINO: 0.683 -> 0.676 | Hurt |
| Add semantic matching guidance (ablation) | DreamBooth with semantic matching and semantic-consistent mask · ViCo dataset | IDINO: 0.676 -> 0.680 | Helped |
| Add semantic matching guidance (ablation) | DreamBooth with semantic matching and semantic-consistent mask · ViCo dataset | TCLIP: 0.232 -> 0.231 | Hurt |
| Replace target keys and values with reference keys and values (ablation) | DreamBooth · ViCo dataset | TCLIP: 0.237 -> 0.201 | Hurt |
Surprising findings
- Plain key-value replacement (MasaCtrl) gives the highest subject scores (IDINO 0.728) but sharply lowers prompt fidelity (TCLIP 0.201), and it produces static subject images that fail to preserve the target structure.
Limitations
- Relies on the quality of the underlying personalized model; fidelity gains depend on the chosen baseline.
- TCLIP prompt-fidelity slightly drops versus some baselines, which the authors attribute to CLIP's imperfect text-image alignment.
Experimental setup
Datasets: ViCo dataset (16 concepts, 31 prompts)
Benchmarks: ViCo dataset evaluation setting challenging non-rigid prompt set (large displacement, occlusion, novel-view)
Metrics: DINO image similarity (IDINO) CLIP image similarity (ICLIP) CLIP text-image similarity (TCLIP) user study ranking
Baselines: Textual Inversion DreamBooth CustomDiffusion FreeU MagicFusion ViCo MasaCtrl
Builds on: Textual Inversion DreamBooth CustomDiffusion MasaCtrl DDIM inversion
Closest papers by meaning
- DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation (ICLR 2025 · similarity 0.87)
- AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generation (NeurIPS 2024 · similarity 0.87)
- PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction (ICLR 2025 · similarity 0.87)
- DreamSteerer: Enhancing Source Image Conditioned Editability using Personalized Diffusion Models (NeurIPS 2024 · similarity 0.87)
- DreamRelation: Bridging Customization and Relation Generation (CVPR 2025 · similarity 0.87)
- Attention Calibration for Disentangled Text-to-Image Personalization (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.