Research map / Video / Video generation
Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Teller is a two-stage streaming audio-driven talking-head framework that uses an autoregressive transformer over RVQ facial-motion tokens (conditioned on Whisper audio) plus a single-step Efficient Temporal Module for body/accessory refinement, reaching up to 25 FPS.
Problem
Diffusion-based audio-driven portrait animation is too slow for real-time use and often produces stiff facial and body/accessory motion (e.g., neck, earrings).
Approach
Model facial motion as discrete RVQ tokens generated autoregressively from streaming audio in 200ms chunks, predicting token pairs per step, then refine with a single-pass temporal attention module using region masks on body parts and accessories instead of iterative diffusion denoising.
Key results
- On HDTF, Teller reports FVD 173.463 (lowest) and Sync-C 7.696 / Sync-D 7.536, with 0.92s per second of video versus 20.93s for Hallo.
- On RAVDESS, Teller reports the lowest FVD of 429.288 and the highest Sync-C of 4.496 among compared methods.
- Real-time analysis on 4 H800 GPUs: total stage-wise cost under 200ms per 200ms audio chunk, reaching up to 25 FPS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Teller instead of Hallo | Audio-driven portrait animation · HDTF | FVD: 174.191 -> 173.463 | Helped |
| Use Teller instead of Hallo | Audio-driven portrait animation · HDTF | FID: 20.639 -> 21.352 | Hurt |
| Use Teller instead of Hallo | Audio-driven portrait animation · HDTF | Time: 20.93s -> 0.92s | Helped |
| Use Teller instead of Hallo | Audio-driven portrait animation · RAVDESS | FVD: 537.478 -> 429.288 | Helped |
| Replace Whisper with funcodec as the audio condition encoder (ablation) | Teller · HDTF | Sync-C: 7.696 -> 4.286 | Hurt |
| Replace Whisper with funcodec as the audio condition encoder (ablation) | Teller · HDTF | Sync-D: 7.536 -> 10.373 | Hurt |
| Use a single-head rather than multi-head architecture (ablation) | Teller · HDTF | Sync-C: 7.696 -> 7.790 | Helped |
| Use a single-head rather than multi-head architecture (ablation) | Teller · HDTF | FVD: 173.463 -> 172.553 | Hurt |
| Add the Efficient Temporal Module (ETM) (ablation) | Teller Stage 1 · Portrait animation body parts and accessories | Physical consistency | Helped |
| Vary the number of frames and tokens in the RVQ module (ablation) | Teller · RVQ face-shape accuracy | Reconstruction loss: 1frames 32tokens, loss:2.4e-6; 12frames 32tokens, loss:14.9e-6 | Mixed |
Surprising findings
- Single-head variant slightly outperforms the multi-head design on FID and Sync metrics (FID 22.110 vs 21.352 for multi-head is worse, Sync-C 7.790 vs 7.696), contrary to the chosen design.
Limitations
- Human evaluation is based on 30 participants and 100 videos, with subjective Likert ratings.
- Comparison of Sync and FID is mixed: Teller's FID is not best on HDTF (21.352 vs 20.639 for Hallo) or RAVDESS (20.352 vs 19.826 for Hallo).
Experimental setup
Datasets: AVSpeech VFHQ HDTF RAVDESS
Benchmarks: HDTF RAVDESS
Metrics: FID FVD Sync-C Sync-D inference time
Baselines: SadTalker EchoMimic AniPortrait Hallo FaceVid2Vid LivePortrait
Builds on: LivePortrait Whisper VQ-VAE residual quantization BERT
Closest papers by meaning
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.90)
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.89)
- Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis (ICLR 2024 · similarity 0.88)
- FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models (CVPR 2024 · similarity 0.88)
- Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style (AAAI 2024 · similarity 0.88)
- Sonic: Shifting Focus to Global Audio Perception in Portrait Animation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.