Research map / Video / Video generation
Multi-subject Open-set Personalization in Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Video Alchemist is a latent Diffusion Transformer with per-subject cross-attention over reference images and entity-word embeddings, trained on an automatically curated dataset with augmentations, enabling multi-subject open-set video personalization without test-time optimization, plus the MSRVTT-Personalization benchmark.
Problem
Video personalization is limited to closed domains, needs per-subject test-time optimization, or supports only a single subject; open-set multi-subject personalization of foreground objects and backgrounds in video is underexplored, and no suitable evaluation exists.
Approach
A DiT with an added cross-attention layer for personalization embeddings that fuse each reference image's tokens with its subject-level entity word, trained on frames sampled from videos with segmented subjects and heavy reference-image augmentation to avoid copy-and-paste overfitting.
Key results
- On MSRVTT-Personalization subject mode, Video Alchemist reaches subject similarity 0.617 with a single reference, 23.2% higher than VideoBooth.
- In face mode, Video Alchemist achieves face similarity 0.382 vs. 0.269 for IP-Adapter (11.3% higher as reported).
- User study: Video Alchemist preferred in 96.5% (quality) and 98.1% (fidelity) of subject-mode comparisons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Video Alchemist instead of VideoBooth | single-reference subject personalization · MSRVTT-Personalization subject mode | Subj-S: 0.395 -> 0.617 | Helped |
| add multiple reference images instead of using a single reference | Video Alchemist · MSRVTT-Personalization subject mode | Subj-S: 0.617 -> 0.626 | Helped |
| add a background reference image | Video Alchemist with multiple subject references · MSRVTT-Personalization subject mode | Vid-S: 0.743 -> 0.780 | Helped |
| add a background reference image | Video Alchemist with multiple subject references · MSRVTT-Personalization subject mode | Text-S: 0.268 -> 0.254 | Hurt |
| add multiple reference images instead of using a single reference | Video Alchemist · MSRVTT-Personalization face mode | Face-S: 0.382 -> 0.411 | Helped |
| use CLIP instead of DINOv2 as the image encoder (ablation) | Video Alchemist · MSRVTT-Personalization subject mode | Text-S: 0.257 -> 0.269 | Helped |
| use CLIP instead of DINOv2 as the image encoder (ablation) | Video Alchemist · MSRVTT-Personalization subject mode | Subj-S: 0.600 -> 0.569 | Hurt |
| remove the corresponding entity-word tokens from image conditioning (ablation) | Video Alchemist · MSRVTT-Personalization subject mode | Subj-S: 0.600 -> 0.566 | Hurt |
| remove the corresponding entity-word tokens from image conditioning (ablation) | Video Alchemist · MSRVTT-Personalization subject mode | Text-S: 0.257 -> 0.256 | Hurt |
| remove reference-image augmentations (ablation) | Video Alchemist · MSRVTT-Personalization subject mode | Dync-D: 0.570 -> 0.506 | Hurt |
| remove reference-image augmentations (ablation) | Video Alchemist · MSRVTT-Personalization subject mode | Text-S: 0.257 -> 0.251 | Hurt |
| remove reference-image augmentations (ablation) | Video Alchemist · MSRVTT-Personalization subject mode | Subj-S: 0.600 -> 0.609 | Helped |
Surprising findings
- Without data augmentation the model reaches higher subject similarity but suffers copy-and-paste behavior, lowering dynamic degree and text similarity.
Limitations
- More reference images sometimes reduce text alignment due to limited flexibility.
- Without augmentation the model replicates reference expressions and poses (copy-and-paste effect).
Experimental setup
Datasets: large-scale video-caption training dataset (curated) MSR-VTT
Benchmarks: MSRVTT-Personalization
Metrics: text similarity video similarity subject similarity face similarity dynamic degree user preference
Baselines: ELITE VideoBooth DreamVideo IP-Adapter PhotoMaker Magic-Me
Builds on: DiT IP-Adapter rectified flow SAM GroundingDINO
Closest papers by meaning
- VideoMage: Multi-Subject and Motion Customization of Text-to-Video Diffusion Models (CVPR 2025 · similarity 0.88)
- DynASyn: Multi-Subject Personalization Enabling Dynamic Action Synthesis (AAAI 2025 · similarity 0.87)
- Movie Weaver: Tuning-Free Multi-Concept Video Personalization with Anchored Prompts (CVPR 2025 · similarity 0.87)
- CustomCrafter: Customized Video Generation with Preserving Motion and Concept Composition Abilities (AAAI 2025 · similarity 0.85)
- OmniVCus: Feedforward Subject-driven Video Customization with Multimodal Control Conditions (NeurIPS 2025 · similarity 0.85)
- PolyVivid: Vivid Multi-Subject Video Generation with Cross-Modal Interaction and Enhancement (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.