Research map / Image generation / Text-to-image diffusion
Resolving Multi-Condition Confusion for Finetuning-Free Personalized Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends IP-Adapter's decoupled cross-attention with attention-based weighted merging of multiple reference-image features, then continues training on a 100K SA-1B multi-object subset selected by a CLIP-based object quality score.
Problem
Finetuning-free personalized text-to-image generation with decoupled cross-attention suffers from "object confusion" when multiple reference images are given, assigning reference features to the wrong objects.
Approach
Estimate per-position object relevance from the original UNet cross-attention between object text tokens and latent features, use it to weight-merge each reference image's features into all positions of the latent, add a learned text-weight predictor, and train on a quality-filtered multi-object SA-1B subset.
Key results
- On Concept101 multi-object generation, Ours (sdxl_plus) reaches CLIP-T 0.7765, CLIP-I 0.6950, DINO 0.4397, outperforming SSR-Encoder (0.7363/0.6895/0.3970) and MC^2 on CLIP-I and DINO scores.
- On DreamBooth multi-object generation, Ours (sdxl) gets CLIP-I 0.726 vs 0.696 for Subject-Diffusion, but lower DINO (0.482 vs 0.506).
- On single-object DreamBooth, the framework improves IP-Adapter across all three metrics (e.g., CLIP-I 0.809 to 0.812, DINO 0.608 to 0.620).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with the weighted-merge framework | SDXL-based method · Concept101 | CLIP-T: 0.7670 -> 0.7765 | Helped |
| Train with the weighted-merge framework | SDXL-based method · DreamBooth multi-object | CLIP-I: 0.726 vs. 0.696 | Helped |
| Train with the weighted-merge framework | IP-Adapter · DreamBooth single-object | CLIP-I: 0.809 -> 0.812 | Helped |
| Train with the weighted-merge framework | ELITE · DreamBooth single-object | DINO: 0.605 -> 0.622 | Helped |
| Use relevance-weighted merging instead of uniform merging (ablation) | Pre-trained IP-Adapter · Concept101 | S_object_relevance: 1.33 -> 1.66 | Helped |
| Add image relevance weighting (ablation) | SDXL weighted-merge training framework · Concept101 | CLIP-I: 0.6851 -> 0.6940 | Helped |
| Add text-feature weighting (ablation) | SDXL weighted-merge training framework · Concept101 | CLIP-I: 0.6851 -> 0.6924 | Helped |
| Use image relevance weighting and text-feature weighting together (ablation) | SDXL weighted-merge training framework · Concept101 | DINO: 0.4079 -> 0.4127 | Helped |
| Add reference features locally rather than uniformly (ablation) | SDXL weighted-merge training framework · Concept101 | CLIP-T: 0.7702 -> 0.7732 | No clear effect |
| Select 100,000 images with highest S_object_quality (ablation) | SDXL weighted-merge training framework · Concept101 | CLIP-T: 0.7733 -> 0.7750 | Helped |
| Select 100,000 images with lowest S_object_pair (ablation) | SDXL weighted-merge training framework · Concept101 | CLIP-T: 0.7733 -> 0.7708 | Hurt |
Surprising findings
- Training-free weighted merge alone raises IP-Adapter's Concept101 DINO from 0.3481 to 0.3624 without any training.
- Locally adding reference features to masked regions gives little improvement over uniform adding.
Limitations
- Dataset construction is only described as 'will be made publicly available', with no release link.
- Multi-object results on DreamBooth are mixed: DINO is below Subject-Diffusion.
- Concept101 baselines marked as re-implemented; original evaluation datasets are not public, limiting direct comparability.
Experimental setup
Datasets: SA-1B Concept101 DreamBooth
Benchmarks: Concept101 DreamBooth
Metrics: CLIP-T CLIP-I DINO object relevance score
Baselines: DreamBooth Custom Diffusion Mix-of-Show MC^2 FastComposer λ-ECLIPSE ELITE IP-Adapter SSR-Encoder Subject-Diffusion Textual Inversion Break-A-Scene
Builds on: IP-Adapter Subject-Diffusion Custom Diffusion Grounding DINO SAM
Closest papers by meaning
- JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation (CVPR 2024 · similarity 0.87)
- DEFT: Decompositional Efficient Fine-Tuning for Text-to-Image Models (NeurIPS 2025 · similarity 0.85)
- Novel Object Synthesis via Adaptive Text-Image Harmony (NeurIPS 2024 · similarity 0.85)
- Personalized Representation from Personalized Generation (ICLR 2025 · similarity 0.84)
- TFCustom: Customized Image Generation with Time-Aware Frequency Feature Guidance (CVPR 2025 · similarity 0.84)
- PatchDPO: Patch-level DPO for Finetuning-free Personalized Image Generation (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.