Research map / Video / Video generation
OmniVCus: Feedforward Subject-driven Video Customization with Multimodal Control Conditions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoCus-Factory to build multi-subject and control-signal training pairs from raw videos, and OmniVCus, a 5B DiT for feedforward subject-driven video customization with depth/mask/camera control, using Lottery Embedding and Temporally Aligned Embedding.
Problem
Feedforward subject-driven video customization is mostly single-subject, struggles to scale to more subjects at inference, and lacks flexible control via depth, mask, camera and text editing signals due to missing multi-subject and control training pairs.
Approach
Builds training pairs from raw unlabeled videos via MLLM captioning (Kosmos-2), SAM2 segmentation, and background-placement augmentation; uses Lottery Embedding to activate more frame embeddings than training subjects and Temporally Aligned Embedding to share frame position embeddings between control and noise tokens; transfers image-editing ability to video through Image-Video Transfer Mixed trai
Key results
- On single-subject customization, OmniVCus reaches CLIP-T 0.3293 and DINO-I 0.5215 vs. best baseline SkyReels 0.2820 and 0.4612.
- On multi-subject customization, DINO-I 0.4965 vs. SkyReels 0.4107, and the model trained with 2 subjects composes 4 subjects at inference.
- User study with 37 participants shows OmniVCus preferred over baselines by large margins (e.g., 91.9% alignment vs. VideoBooth in one track).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add subject filtering (ablation) | VideoCus-Factory · Single-subject customization | DINO-I: 0.2405 -> 0.5053; 0.2648 | Helped |
| add data augmentation (ablation) | VideoCus-Factory · Single-subject customization | CLIP-T: 0.2431 -> 0.3293; 0.0862 | Helped |
| add data augmentation (ablation) | VideoCus-Factory · Single-subject customization | Dynamic: 0.3826 -> 0.5541; 0.1715 | Helped |
| use temporally aligned embeddings instead of naive separate frame embeddings (ablation) | OmniVCus · Depth-controlled customization | CLIP-T: 0.2618 -> 0.3054 | Helped |
| use temporally aligned embeddings instead of naive separate frame embeddings (ablation) | OmniVCus · Depth-controlled customization | DINO-I: 0.2947 -> 0.3794 | Helped |
| add depth tokens to noise after an MLP (ablation) | OmniVCus · Depth-controlled customization | CLIP-T: 0.2618 -> 0.1722 | Hurt |
| add depth tokens to noise after an MLP (ablation) | OmniVCus · Depth-controlled customization | DINO-I: 0.2947 -> 0.1680 | Hurt |
| use Lottery Embedding (ablation) | OmniVCus · Multi-subject customization | CLIP-T: 0.2105 -> 0.2728; 0.0623 | Helped |
| use Lottery Embedding (ablation) | OmniVCus · Multi-subject customization | DINO-I: 0.3364 -> 0.4163; 0.0799 | Helped |
| use no mixed training instead of IVTM (ablation) | OmniVCus · Instructive editing customization | CLIP-T: 0.3126 -> 0.2137 | Hurt |
| directly mix training with OmniEdit data instead of IVTM (ablation) | OmniVCus · Instructive editing customization | CLIP-T: 0.3126 -> 0.2585 | Hurt |
| use OmniVCus instead of SkyReels | single-subject customization · Single-subject customization | CLIP-T: 0.2820 -> 0.3293 | Helped |
Surprising findings
- Add-to-noise control embedding causes model collapse (CLIP-T 0.1722, DINO-I 0.1680).
Limitations
- Training data comes from an internal video pool that is not released.
- Evaluation uses small self-collected sets and a 37-participant user study.
- Multi-subject results are averaged over double/triple subjects only because a baseline supports at most three subjects.
Experimental setup
Datasets: internal video data pool Objaverse OmniEdit text-to-image data text-to-video data
Benchmarks: internal evaluation set (112 single-subject, 76/74/56 double/triple/quadruple, 112 camera-controlled samples)
Metrics: CLIP-T CLIP-I DINO-I temporal consistency dynamic degree user preference
Baselines: VideoBooth DreamVideo Wan2.1-I2V SkyReels-A2 MotionCtrl CameraCtrl CamI2V
Builds on: DiT flow matching Kosmos-2 SAM2 OmniEdit
Closest papers by meaning
- Multi-subject Open-set Personalization in Video Generation (CVPR 2025 · similarity 0.85)
- CustomCrafter: Customized Video Generation with Preserving Motion and Concept Composition Abilities (AAAI 2025 · similarity 0.85)
- PolyVivid: Vivid Multi-Subject Video Generation with Cross-Modal Interaction and Enhancement (NeurIPS 2025 · similarity 0.85)
- VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models (CVPR 2024 · similarity 0.85)
- VideoMage: Multi-Subject and Motion Customization of Text-to-Video Diffusion Models (CVPR 2025 · similarity 0.84)
- DreamVideo: Composing Your Dream Videos with Customized Subject and Motion (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.