Research map / Video / Video generation
EchoShot: Multi-Shot Portrait Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds EchoShot, a Wan-1.3B-based multi-shot portrait video generator using shot-aware RoPE (TcRoPE, TaRoPE) trained on the new PortraitGala dataset, with extensions for reference-based and infinite-shot generation.
Problem
Single-shot video diffusion models cannot generate multi-shot portrait videos with consistent identity and fine-grained per-shot text control; existing workarounds (iterative personalization, keyframe-to-video) suffer from prompt-adherence trade-offs, keyframe bottlenecks and identity drift.
Approach
Modify a pretrained video DiT with two training-free-parameter RoPE variants: TcRoPE adds a rotary phase shift per shot in self-attention to mark shot boundaries, and TaRoPE applies shot-indexed rotations in video-text cross-attention to suppress attention between mismatched shot-caption pairs, enabling native multi-shot training on a many-to-one identity dataset.
Key results
- EchoShot-1B reaches 73.74% FaceSim-Arc and 69.43% FaceSim-Cur, the highest among compared methods in the MT2V task.
- EchoShot-1B scores 95.84% App., 88.86% Mot. and 94.72% Bg. prompt controllability, well above baselines such as IC-LoRA+Wan-14B (87.59, 83.71, 75.37).
- In the user study with 20 participants, EchoShot wins at least 70% of matchups for identity consistency and prompt controllability.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use native multi-shot EchoShot instead of StoryDiffusion+Wan-14B | MT2V · MT2V | FaceSim-Arc: 68.65 -> 73.74 | Helped |
| Use native multi-shot EchoShot instead of IC-LoRA+Wan-14B | MT2V · MT2V | App.: 87.59 -> 95.84 | Helped |
| Use native multi-shot EchoShot instead of HunyuanVideo-13B (2 shots) | MT2V · MT2V | FaceSim-Cur: 61.25 -> 69.43 | Helped |
| Add TcRoPE to simple concatenation with vanilla RoPE (ablation) | EchoShot on the reduced-scale MT2V dataset · MT2V reduced-scale dataset | App.: 55.79 -> 81.72 | Helped |
| Add TcRoPE to simple concatenation with vanilla RoPE (ablation) | EchoShot on the reduced-scale MT2V dataset · MT2V reduced-scale dataset | Mot.: 59.16 -> 72.84 | Helped |
| Add TcRoPE to simple concatenation with vanilla RoPE (ablation) | EchoShot on the reduced-scale MT2V dataset · MT2V reduced-scale dataset | Bg.: 47.56 -> 74.22 | Helped |
| Add TaRoPE to SC+TcRoPE (ablation) | EchoShot on the reduced-scale MT2V dataset · MT2V reduced-scale dataset | App.: 81.72 -> 94.12 | Helped |
| Add TaRoPE to SC+TcRoPE (ablation) | EchoShot on the reduced-scale MT2V dataset · MT2V reduced-scale dataset | Mot.: 72.84 -> 87.41 | Helped |
| Add TaRoPE to SC+TcRoPE (ablation) | EchoShot on the reduced-scale MT2V dataset · MT2V reduced-scale dataset | Bg.: 74.22 -> 93.96 | Helped |
| Set phase shift scale j=2 instead of j=4 (ablation) | EchoShot · MT2V reduced-scale dataset | FaceSim-Arc: 75.83 -> 75.45 | No clear effect |
| Set phase shift scale j=6 instead of j=4 (ablation) | EchoShot · MT2V reduced-scale dataset | FaceSim-Arc: 75.83 -> 70.19 | Hurt |
| Set mismatch suppression scale k=2 instead of k=6 (ablation) | EchoShot · MT2V reduced-scale dataset | App.: 94.12 -> 84.20 | Hurt |
Surprising findings
- Simply concatenating shot captions (SC) yields the highest FaceSim-Arc among ablations, suggesting identity consistency is not what the shot-aware mechanisms mainly improve.
- A 1B-parameter model outperforms 13B and 14B baselines on the reported metrics.
Limitations
- Cannot directly extend shots from previous content, limiting longer continuous single-shot segments.
- Multi-subject multi-shot scripts are not supported.
- Evaluation relies on a VLM scorer and a single 100-prompt benchmark plus a 20-person user study.
Experimental setup
Datasets: PortraitGala CelebV-HQ CelebV-Text OpenVid-1M OpenHumanVid
Benchmarks: EchoShot MT2V benchmark (100 LLM-generated multi-shot prompt sets)
Metrics: FaceSim-Arc FaceSim-Cur VLM-scored appearance, motion, background alignment VLM-scored static and dynamic quality user study win rate
Baselines: StoryDiffusion+Wan-I2V IC-LoRA+Wan-I2V HunyuanVideo ConsisID Kling
Builds on: Wan2.1 rectified flow 3D RoPE PySceneDetect
Closest papers by meaning
- ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Models (CVPR 2025 · similarity 0.88)
- EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditions (AAAI 2025 · similarity 0.88)
- RealPortrait: Realistic Portrait Animation with Diffusion Transformers (AAAI 2025 · similarity 0.88)
- VideoMage: Multi-Subject and Motion Customization of Text-to-Video Diffusion Models (CVPR 2025 · similarity 0.87)
- FaceShot: Bring Any Character into Life (ICLR 2025 · similarity 0.87)
- Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.