Research map / Video / Video generation
PolyVivid: Vivid Multi-Subject Video Generation with Cross-Modal Interaction and Enhancement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
PolyVivid is a multi-subject video customization framework that uses a VLLM (LLaVA) text-image fusion, 3D-RoPE text-image interaction, attention-inherited identity injection on HunyuanVideo, and an MLLM-based clique-consolidated data pipeline.
Problem
Multi-subject video customization struggles to ground subject images to the correct textual entities, keep identities consistent across multiple subjects, and handle ambiguous subject data during construction.
Approach
Embed subject images into the text space via a VLLM with structured identity prompts, bind text and image tokens with a text-image 3D-RoPE for bidirectional fusion, and inject identity features via a LoRA-reparameterized MM-attention cross-attention that treats all frames equally.
Key results
- PolyVivid achieves Face-sim 0.642 and DINO-sim 0.623, the highest among compared methods (best baseline Face-sim 0.554 for SkyReels-A2).
- PolyVivid obtains the lowest FVD of 959.74 versus 980.49 for Pika (second best).
- Ablation shows the identity injection module raises Face-sim from 0.601 to 0.642 and reduces FVD from 965.62 to 959.74.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add text-image interaction to LLaVA fusion (ablation) | PolyVivid · Multi-subject video customization test set | Face-sim: 0.381 -> 0.584 | Helped |
| add text-image interaction to LLaVA fusion (ablation) | PolyVivid · Multi-subject video customization test set | CLIP-B: 0.345 -> 0.330 | Hurt |
| add text-image interaction 3D-RoPE (ablation) | PolyVivid with LLaVA fusion and text-image interaction · Multi-subject video customization test set | Face-sim: 0.584 -> 0.601 | Helped |
| add text-image interaction 3D-RoPE (ablation) | PolyVivid with LLaVA fusion and text-image interaction · Multi-subject video customization test set | CLIP-B: 0.330 -> 0.340 | Helped |
| add attention-inherited identity injection (ablation) | PolyVivid with LLaVA, text-image interaction and 3D-RoPE · Multi-subject video customization test set | Face-sim: 0.601 -> 0.642 | Helped |
| add attention-inherited identity injection (ablation) | PolyVivid with LLaVA, text-image interaction and 3D-RoPE · Multi-subject video customization test set | FVD: 965.62 -> 959.74 | Helped |
| replace adapter-based injection with token concatenation (ablation) | LLaVA-based condition injection · Multi-subject video customization test set | Face-sim: 0.401 -> 0.628 | Helped |
| replace adapter-based injection with token concatenation (ablation) | LLaVA-based condition injection · Multi-subject video customization test set | CLIP-B: 0.338 -> 0.328 | Hurt |
| use PolyVivid rather than the strongest baseline | Multi-subject video customization test set · Multi-subject video customization test set | Face-sim: 0.554 -> 0.642 | Helped |
| use PolyVivid rather than the strongest baseline | Multi-subject video customization test set · Multi-subject video customization test set | DINO-sim: 0.619 -> 0.623 | Helped |
| use PolyVivid rather than the strongest baseline | Multi-subject video customization test set · Multi-subject video customization test set | CLIP-L: 0.282 -> 0.281 | Hurt |
| use PolyVivid rather than the strongest baseline | Multi-subject video customization test set · Multi-subject video customization test set | FVD: 980.49 -> 959.74 | Helped |
Surprising findings
- Token concatenation and adapter-based injection both underperform the proposed injection, with adapter-based injection giving only marginal identity gains over LLaVA alone.
Limitations
- Quality and controllability bounded by the underlying video backbone and pretrained MLLM/VAE.
- Struggles with highly complex scenes with many subjects, intricate backgrounds, or fine-grained physical reasoning.
- MLLM-based grounding/segmentation may err under occlusion or ambiguous cues.
Experimental setup
Datasets: Panda-70M Koala-36M self-collected videos
Benchmarks: 100-pair custom multi-subject test set
Metrics: Face-sim DINO-sim CLIP-B CLIP-L FVD Temporal consistency
Baselines: Vidu-2.0 Keling-1.6 Pika Hailuo SkyReels-A2 VACE-1.3B
Builds on: HunyuanVideo LLaVA IP-Adapter OmniControl LoRA
Closest papers by meaning
- VideoMage: Multi-Subject and Motion Customization of Text-to-Video Diffusion Models (CVPR 2025 · similarity 0.87)
- ARGUS: Stacked Multi-View Identity Mosaic Injection for Subject-Preserving Video Generation (NeurIPS 2026 · similarity 0.87)
- FaithfulFaces: Pose-Faithful Facial Identity Preservation for Text-to-Video Generation (NeurIPS 2026 · similarity 0.86)
- XVerse: Consistent Multi-Subject Control of Identity and Semantic Attributes via DiT Modulation (NeurIPS 2025 · similarity 0.85)
- Multi-subject Open-set Personalization in Video Generation (CVPR 2025 · similarity 0.85)
- CustomCrafter: Customized Video Generation with Preserving Motion and Concept Composition Abilities (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.