Research map / Image generation / Text-to-image diffusion
DynASyn: Multi-Subject Personalization Enabling Dynamic Action Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DynASyn, a single-image multi-subject personalization method for Stable Diffusion that uses concept-based attention regularization and GPT-4-driven prompt-and-image augmentation (SDE-based editing) to enable dynamic actions and interactions without overfitting.
Problem
Multi-subject personalization from a single reference image tends to overfit to the reference, making it hard to change subject behaviors/actions or model dynamic interactions in novel contexts.
Approach
Regularize cross-attention between subject placeholder tokens and images using attention maps from class-name prompts (concept priors), and augment training with LLM-generated action prompts and SDE-guided (SDEdit-style) augmented images that are aligned to the original subject identity.
Key results
- On action sentences, DynASyn achieves IR 0.827 and CLIP-T 0.359, versus 0.604 and 0.31 for Break-a-Scene.
- User study on interaction sentences gives DynASyn overall 7.7 vs 6.2 for Break-a-Scene.
- Ablation shows adding action component raises IR from 0.616 to 0.829 with appearance component only, while CLIP-I stays around 0.76-0.79.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DynASyn instead of Break-a-Scene | Multi-subject personalization · Plain Sentence | CLIP-T: 0.351 -> 0.376 | Helped |
| Use DynASyn instead of Break-a-Scene | Multi-subject personalization · Action Sentence | IR: 0.604 -> 0.827 | Helped |
| Use DynASyn instead of Break-a-Scene | Multi-subject personalization · Interaction Sentence | CLIP-T: 0.296 -> 0.346 | Helped |
| Use DreamBooth (+mask) instead of DynASyn | Multi-subject personalization · Plain Sentence | CLIP-I: 0.84 -> 0.797 | Hurt |
| Use DreamBooth (+mask) instead of DynASyn | Multi-subject personalization · Action Sentence | CLIP-I: 0.79 -> 0.771 | Hurt |
| Use DreamBooth (+mask) instead of DynASyn | Multi-subject personalization · Interaction Sentence | CLIP-I: 0.823 -> 0.731 | Hurt |
| Add appearance alignment (App) (ablation) | DynASyn with masked diffusion only · Ablation evaluation | CLIP-T: 0.292 -> 0.298 | Helped |
| Add appearance alignment (App) (ablation) | DynASyn with masked diffusion only · Ablation evaluation | CLIP-I: 0.767 -> 0.786 | Helped |
| Add action alignment (Act) (ablation) | DynASyn with masked diffusion only · Ablation evaluation | CLIP-T: 0.292 -> 0.306 | Helped |
| Add action alignment (Act) (ablation) | DynASyn with masked diffusion only · Ablation evaluation | IR: 0.341 -> 0.732 | Helped |
| Add both appearance alignment (App) and action alignment (Act) (ablation) | DynASyn with masked diffusion only · Ablation evaluation | IR: 0.341 -> 0.829 | Helped |
| Add both appearance alignment (App) and action alignment (Act) (ablation) | DynASyn with masked diffusion only · Ablation evaluation | CLIP-I: 0.767 -> 0.767 | No clear effect |
Surprising findings
- DreamBooth attains the highest CLIP-I and user identity scores but the lowest text alignment, indicating CLIP-I rewards overfitting to the input.
Limitations
- Dependent on the quality of the underlying T2I backbone (Stable Diffusion struggles with challenging prompts).
- Relies on prior knowledge from VLM/LLM and T2I models; direct comparison with some recent works was not possible because LAION was taken down.
- Evaluation uses a small set of custom prompts and 15 volunteers for the user study.
Experimental setup
Datasets: Break-a-Scene datasets COCO Zhang et al. datasets proprietary collected datasets
Benchmarks: custom plain/action/interaction prompt sets
Metrics: CLIP-T CLIP-I Image Reward user study overall/text/identity score (1-10)
Baselines: Textual Inversion (+mask) DreamBooth (+mask) Break-a-Scene
Builds on: DreamBooth Textual Inversion Break-a-Scene SDEdit Segment Anything
Closest papers by meaning
- VideoMage: Multi-Subject and Motion Customization of Text-to-Video Diffusion Models (CVPR 2025 · similarity 0.88)
- MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance (ICLR 2025 · similarity 0.87)
- Multi-subject Open-set Personalization in Video Generation (CVPR 2025 · similarity 0.87)
- PortraitBooth: A Versatile Portrait Model for Fast Identity-preserved Personalization (CVPR 2024 · similarity 0.86)
- JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation (CVPR 2024 · similarity 0.86)
- Identity Decoupling for Multi-Subject Personalization of Text-to-Image Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.