Research map / Video / Video generation
Identity-Preserving Text-to-Video Generation by Frequency Decomposition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ConsisID, a tuning-free DiT-based identity-preserving text-to-video model that injects low-frequency facial signals (keypoint-augmented reference image concatenated to latent) and high-frequency identity features (face recognition plus CLIP via Q-Former, cross-attention in blocks), trained with a hierarchical strategy on CogVideoX-5B.
Problem
Identity-preserving text-to-video generation typically requires per-identity finetuning, and existing tuning-free methods are U-Net based, produce only face-region videos, and preserve identity poorly; DiT-based controllable generation is underexplored.
Approach
Decompose facial identity into low-frequency global features (reference image plus facial keypoints concatenated with the noise latent in shallow layers) and high-frequency intrinsic identity features (face recognition backbone plus CLIP fused by Q-Former and injected via cross-attention in each DiT block), motivated by frequency analyses of transformers and diffusion models, and train with coarse
Key results
- ConsisID achieves FaceSim-Arc 0.58 and FaceSim-Cur 0.60 vs. 0.32 and 0.33 for ID-Animator on a 30-identity benchmark, with CLIPScore 27.93 vs. 24.97.
- In a user study of 103 valid questionnaires, ConsisID is preferred over ID-Animator on identity preservation, visual quality, text relevance, and motion amplitude.
- Removing the global facial extractor drops FaceSim-Arc to 0.05, while the full model reaches 0.73 FaceSim-Arc and 36.77 CLIPScore in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the Global Facial Extractor (low-frequency reference-face and key-point signal) (ablation) | ConsisID · IPT2V evaluation | FaceSim-Arc: 0.05 -> 0.73 | Helped |
| Add the Local Facial Extractor (high-frequency identity features) (ablation) | ConsisID · IPT2V evaluation | FaceSim-Arc: 0.66 -> 0.73 | Helped |
| Add coarse-to-fine training (ablation) | ConsisID · IPT2V evaluation | FaceSim-Arc: 0.54 -> 0.73 | Helped |
| Add dynamic mask loss (ablation) | ConsisID · IPT2V evaluation | FID: 187.78 -> 127.42 | Helped |
| Add dynamic cross-face loss (ablation) | ConsisID · IPT2V evaluation | CLIPScore: 32.21 -> 36.77 | Helped |
| Inject both high- and low-frequency face signals rather than only high-frequency signals (ablation) | ConsisID · IPT2V evaluation | FaceSim-Arc: 0.05 -> 0.73 | Helped |
| Add low-frequency face signals alongside high-frequency signals (ablation) | ConsisID · IPT2V evaluation | FaceSim-Arc: 0.66 -> 0.73 | Helped |
| Include facial key points in the low-frequency signal (ablation) | ConsisID · IPT2V evaluation | FaceSim-Arc: 0.64 -> 0.73 | Helped |
| Inject high-frequency identity signals inside the attention block rather than at its output (ablation) | ConsisID · IPT2V evaluation | FaceSim-Arc: 0.62 -> 0.73 | Helped |
Surprising findings
- Injecting only high-frequency identity signals causes failure to converge, and injecting identity signals at the attention block input causes gradient explosion.
- Removing the global facial extractor drops FaceSim-Arc to 0.05, showing the low-frequency path is essential for training convergence.
Limitations
- Benchmark is small (30 identities, 90 prompts) and the training dataset is in-house and unreleased.
- Motion amplitude is evaluated only through user study because reliable metrics are lacking.
- FID is reported only as a reference due to weak alignment with human perception.
Experimental setup
Datasets: in-house human-centric video dataset
Benchmarks: in-house benchmark of 30 identities x 5 images with 90 prompts
Metrics: FaceSim-Arc FaceSim-Cur CLIPScore FID user preference
Baselines: ID-Animator MovieGen
Builds on: CogVideoX IP-Adapter InstantID ID-Animator DPM-Solver
Closest papers by meaning
- FaithfulFaces: Pose-Faithful Facial Identity Preservation for Text-to-Video Generation (NeurIPS 2026 · similarity 0.89)
- StableAnimator: High-Quality Identity-Preserving Human Image Animation (CVPR 2025 · similarity 0.86)
- ARGUS: Stacked Multi-View Identity Mosaic Injection for Subject-Preserving Video Generation (NeurIPS 2026 · similarity 0.86)
- FaceA-Net: Facial Attribute-Driven ID Preserving Image Generation Network (AAAI 2025 · similarity 0.85)
- Diffusion Self-Distillation for Zero-Shot Customized Image Generation (CVPR 2025 · similarity 0.85)
- Frame In-N-Out: Unbounded Controllable Image-to-Video Generation (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.