Research map / Image generation / Text-to-image diffusion
Multi-party Collaborative Attention Control for Image Customization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MCA-Ctrl, a tuning-free image customization method that coordinates three parallel Stable Diffusion processes via self-attention local query and global injection, plus a DINO+SAM Subject Localization Module for complex scenes.
Problem
Image customization with text or image conditions suffers from fine-tuning cost, subject leakage/confusion in complex visual scenes, inconsistent backgrounds under image conditions, and limited task coverage.
Approach
Coordinates three parallel diffusion processes (subject, condition, target) inside self-attention layers using two complementary operations: SALQ (target queries subject foreground and condition background) and SAGI (injects subject and condition attention features globally), with a Subject Localization Module (DINO + SAM) to refine high-response regions. No fine-tuning is needed.
Key results
- On DreamEditBench subject swapping, Ours (Specified) reaches DINOsub 0.6433 and CLIP-Isub 0.8113, versus 0.6155 and 0.8009 for BLIP-Diffusion.
- On DreamBench subject generation, Ours (Specified) attains DINO 0.6724 and CLIP-I 0.8441, above BLIP-Diffusion (0.6700, 0.8250) and DreamBooth (0.6680, 0.8430) on DINO and CLIP-I respectively.
- Human evaluation on DreamBench gives Ours (Specified) an overall score of 2.73, the highest among compared methods (BLIP-Diffusion 2.6, IP-Adapter 2.63).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use class-specific attention parameters instead of uniform parameters | MCA-Ctrl · DreamEditBench subject swapping | DINOsub: 0.6327±0.004 -> 0.6433±0.005 | Helped |
| Use class-specific attention parameters instead of uniform parameters | MCA-Ctrl · DreamBench subject generation | ImageReward: 0.3037±0.05 -> 0.4132±0.06 | Helped |
| Remove SALQ (ablation) | MCA-Ctrl · DreamEditBench subject swapping | DINOsub: 0.6327 -> 0.4238 | Hurt |
| Remove SAGI (ablation) | MCA-Ctrl · DreamEditBench subject swapping | ImageReward: 0.2728 -> 0.2716 | Hurt |
| Remove mask dilation (ablation) | MCA-Ctrl · DreamEditBench subject swapping | DINOsub: 0.6327 -> 0.5611 | Hurt |
| Remove the Subject Localization Module (ablation) | MCA-Ctrl · DreamEditBench subject swapping | DINOsub: 0.6327 -> 0.4914 | Hurt |
| Reverse the SAGI and SALQ execution order (ablation) | MCA-Ctrl · DreamEditBench subject swapping | ImageReward: 0.2728 -> 0.1076 | Hurt |
| Use uniform rather than class-specific parameters | MCA-Ctrl · DreamBench subject generation | DINO: 0.6724±0.004 -> 0.6610±0.002 | Hurt |
Surprising findings
- Ours (Uniform) is slightly below DreamBooth on DINOsub and below DreamBooth and BLIP-Diffusion on some generation metrics, and per-class parameter tuning was needed to surpass them.
- Removing SALQ drops DINOsub from 0.6327 to 0.4238 while CLIP-Iback and DINOback rise, showing a trade-off with background consistency.
Limitations
- Constrained by the base model; struggles with fine-grained subject features such as text.
- Color changes may only affect local subject regions.
- Per-class tuning of SGI/EGI/LayerLQ/ELQ is needed to reach the best reported scores.
Experimental setup
Datasets: DreamBench DreamEditBench
Benchmarks: DreamBench DreamEditBench
Metrics: DINO CLIP-I CLIP-T ImageReward human evaluation score
Baselines: DreamBooth Textual Inversion Re-Imagen BLIP-Diffusion IP-Adapter FreeCustom PHOTOSWAP Customized-DiffEdit DreamEditor
Builds on: MasaCtrl Prompt-to-Prompt DDIM inversion Stable Diffusion
Closest papers by meaning
- Devil is in the Detail: Towards Injecting Fine Details of Image Prompt in Image Generation via Conflict-free Guidance and Stratified Attention (CVPR 2025 · similarity 0.87)
- MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance (ICLR 2025 · similarity 0.86)
- Personalized Image Editing in Text-to-Image Diffusion Models via Collaborative Direct Preference Optimization (NeurIPS 2025 · similarity 0.86)
- Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without Guidance (NeurIPS 2024 · similarity 0.86)
- Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing (CVPR 2024 · similarity 0.86)
- How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization? (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.