Research map / Video / Video generation
Sonic: Shifting Focus to Global Audio Perception in Portrait Animation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Sonic proposes an audio-only talking portrait animation paradigm that uses context-enhanced audio learning, a motion-decoupled controller, and time-aware position shift fusion to achieve global audio perception for long, diverse, and stable videos.
Problem
Audio-driven talking face generation relies on auxiliary visual signals (motion frames, landmarks, spatial priors) that limit motion richness and cause temporal inconsistency, and long-video generation lacks efficient audio-conditioned global context.
Approach
Drive portrait animation solely from audio: long-range intra-clip audio temporal attention, separately controlled head and expression motion buckets predicted from audio and reference image, and a shifted-window inference over denoising timesteps that propagates inter-clip context without motion frames or overlap.
Key results
- On HDTF, Sonic achieves FID 29.104 and FVD 301.173, about 8% lower than the second-best method.
- On CelebV-HQ, Sonic's E-FID of 1.783 is 16.5% lower than the second-best result.
- In a 40-participant user study, Sonic scored 4.58 on lip sync and 4.55 on motion diversity, the highest among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Sonic instead of the strongest baseline | Talking-face generation · HDTF test | FID: 30.176 -> 29.104 | Helped |
| Use Sonic instead of the strongest baseline | Talking-face generation · HDTF test | FVD: 328.540 -> 301.173 | Helped |
| Use Sonic instead of the strongest baseline | Talking-face generation · CelebV-HQ test | FVD: 481.336 -> 483.108 | Hurt |
| Use Sonic instead of the strongest baseline | Talking-face generation · CelebV-HQ test | Sync-C: 2.680 -> 2.689 | Helped |
| Add temporal audio attention (ablation) | Sonic · CelebV-HQ test | Sync-C: 2.610 -> 2.689 | Helped |
| Add temporal audio attention (ablation) | Sonic · CelebV-HQ test | E-FID: 2.311 -> 1.783 | Helped |
| Add motion controller (ablation) | Sonic · CelebV-HQ test | Sync-C: 2.193 -> 2.689 | Helped |
| Add motion controller (ablation) | Sonic · CelebV-HQ test | Dynamic: 0.22 -> 0.86 | Helped |
| Add time-aware position shift fusion (ablation) | Sonic · CelebV-HQ test | Sync-C: 2.509 -> 2.689 | Helped |
| Add time-aware position shift fusion (ablation) | Sonic · CelebV-HQ test | Smooth: 0.9969 -> 0.9972 | Helped |
| Increase dynamic scale to β = 2.0 instead of β = 1.0 (ablation) | Sonic motion controller · CelebV-HQ test | Dynamic: 0.86 -> 0.98 | Helped |
| Increase dynamic scale to β = 2.0 instead of β = 1.0 (ablation) | Sonic motion controller · CelebV-HQ test | Smooth: 0.9972 -> 0.9969 | Hurt |
Surprising findings
- Time-aware shift fusion gives better lip sync and smoothness than motion-frame and overlap strategies while being faster (17.04 s vs 19.70 s and 22.49 s runtime).
Limitations
- Ablation on motion controller and comparisons rely on heuristic metrics (FID, Sync-C/D, VBench dynamic degree) and a small 40-person user study.
- EMO could not be quantitatively compared since it is closed source; only qualitative demo comparison was provided.
Experimental setup
Datasets: VFHQ CelebV-Text VoxCeleb2 HDTF CelebV-HQ
Benchmarks: HDTF CelebV-HQ
Metrics: FID FVD Sync-C Sync-D E-FID F-SIM Smooth Dynamic degree Runtime user study scores
Baselines: SadTalker AniPortrait V-Express Hallo Hallo2 EchoMimic EMO
Builds on: Stable Video Diffusion AnimateDiff Whisper Lumiere
Closest papers by meaning
- KeyFace: Expressive Audio-Driven Facial Animation for Long Sequences via KeyFrame Interpolation (CVPR 2025 · similarity 0.90)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.89)
- Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generation (CVPR 2025 · similarity 0.88)
- SonicVisionLM: Playing Sound with Vision Language Models (CVPR 2024 · similarity 0.87)
- EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion (CVPR 2025 · similarity 0.87)
- FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audio (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.