Research map / Image generation / Text-to-image diffusion
DreamRelation: Bridging Customization and Relation Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DreamRelation, a LoRA-based relation-aware customized image generation method built on MS-Diffusion, using a DALL-E 3 triplet data engine, a keypoint matching loss, and local CLIP token injection, evaluated on a new RelationBench.
Problem
Customized text-to-image generation preserves object identities from image prompts but fails to render the relationships between customized objects specified in text prompts, and suffers from object confusion when objects overlap.
Approach
Disentangle identity from relation learning by fine-tuning LoRA on text cross-attention using relation-specific triplets generated by DALL-E 3 (sharing object identity across images), add a keypoint matching loss in VAE latent space to guide pose, and inject dense local CLIP tokens to distinguish overlapping objects.
Key results
- On RelationBench single-object, DreamRelation reaches CLIP-T 30.6 and CLIP-R 21.4, versus 27.8 and 19.3 for the strongest baseline ReVersion+MS.
- On RelationBench multi-object, DreamRelation reaches CLIP-R 20.4 and DINO 62.1, versus 19.2 and 59.7 for ReVersion+MS.
- Ablation on multi-object RelationBench: removing keypoint matching loss drops DINO from 62.1 to 61.2 and CLIP-R from 20.4 to 19.2; removing local tokens drops CLIP-I from 75.4 to 75.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the relation-aware data engine rather than train on cropped relation-specific images (ablation) | DreamRelation · RelationBench multi-object | CLIP-T: 27.3 -> 28.9 | Helped |
| add the relation-aware data engine rather than train on cropped relation-specific images (ablation) | DreamRelation · RelationBench multi-object | CLIP-R: 19.4 -> 20.4 | Helped |
| add local token injection (ablation) | DreamRelation · RelationBench multi-object | CLIP-I: 75.1 -> 75.4 | Helped |
| add local token injection (ablation) | DreamRelation · RelationBench multi-object | DINO: 59.9 -> 62.1 | Helped |
| add keypoint matching loss (ablation) | DreamRelation · RelationBench multi-object | DINO: 61.2 -> 62.1 | Helped |
| use 16 local tokens instead of 4 or 64 (ablation) | DreamRelation · RelationBench multi-object | CLIP-R: 19.4; 20.4; 18.1 | Mixed |
| set the KML weight to λ=1e-3 instead of λ=1e-2 or λ=1e-4 (ablation) | DreamRelation · RelationBench multi-object | CLIP-T: 27.5; 28.9; 26.8 | Helped |
| use DreamRelation rather than the strongest reported baseline | RelationBench single-object · RelationBench single-object | CLIP-T: 27.8 -> 30.6 | Helped |
| use DreamRelation rather than the strongest reported baseline | RelationBench multi-object · RelationBench multi-object | DINO: 58.8 -> 62.1 | Helped |
| use DreamRelation rather than the strongest reported baseline | DreamBench single-object · DreamBench | CLIP-T: 31.5 -> 31.6 | Helped |
| use DreamRelation rather than the strongest reported baseline | DreamBench single-object · DreamBench | CLIP-I: 82.1 -> 81.7 | Hurt |
| use DreamRelation rather than the strongest reported baseline | M-CustomConcept101 multi-object · M-CustomConcept101 | DINO: 52.2 -> 52.3 | Helped |
Surprising findings
- Single-object CLIP-I is slightly lower than some baselines (e.g., AnyDoor 77.4 vs 77.9 is higher for ours; SSR-Encoder 80.4 is higher), which the authors attribute to large pose variation reducing CLIP-I.
Limitations
- Relation-aware data depends on DALL-E 3 generated triplets, which may limit scope to categories it can keep identity-consistent.
- Keypoint matching relies on X-Pose keypoint detection, which is designed for common objects rather than humans.
- Single-object CLIP-I is slightly lower than some baselines due to pose variation.
Experimental setup
Datasets: DreamBench CustomConcept101
Benchmarks: RelationBench DreamBench M-CustomConcept101
Metrics: CLIP-T CLIP-R CLIP-I DINO
Baselines: DreamBooth Custom Diffusion Cones-V2 ELITE AnyDoor BLIP-Diffusion SSR-Encoder MS-Diffusion ReVersion+MS Mix-of-Show MultiBooth λ-eclipse
Builds on: MS-Diffusion ReVersion DreamBooth SDXL X-Pose
Closest papers by meaning
- Relation Rectification in Diffusion Model (CVPR 2024 · similarity 0.87)
- JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation (CVPR 2024 · similarity 0.87)
- DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization (CVPR 2024 · similarity 0.87)
- RelationAdapter: Learning and Transferring Visual Relation with Diffusion Transformers (NeurIPS 2025 · similarity 0.87)
- Relation-aware Hierarchical Prompt for Open-vocabulary Scene Graph Generation (AAAI 2025 · similarity 0.86)
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.