Research map / Video / Video generation
OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimicking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OmniTalker, a dual-branch DiT with cross-modal MM-Attention fusion that jointly generates talking audio and video from text, mimicking a one-shot reference's speech and facial/head styles via masked in-context infilling, running at 25 FPS.
Problem
Text-driven talking head generation is underexplored; cascaded TTS-plus-audio-driven pipelines suffer from redundancy, error propagation, and audio-visual style mismatch, and existing end-to-end methods need per-identity training.
Approach
A flow-matching dual-branch DiT jointly models mel-spectrogram and visual codes (blendshapes, head pose, eye movements) with MM-Attention fusion in shallow layers and single-modality blocks deeper; style is transferred in-context by masked infilling from a reference audio-video clip, with a GAN warp-based renderer for video.
Key results
- On VoxCeleb2, OmniTalker reaches FVD 102.54 vs. 144.40 for the next best (StyleTalk) and E-FID 0.02 / P-FID 0.04 vs. 0.31 best baseline P-FID.
- On SEED-ZH TTS comparison, WER 1.55% vs. F5-TTS 1.56%, with SIM-A 0.766 vs. MaskGCT 0.774 best.
- Runs at 25 FPS on a single RTX 4090 at 512x512 resolution with a 0.8B-parameter model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use visual condition (α_Cr=2.5) instead of α_Cr=0 (ablation) | OmniTalker · Custom ZH | WER(%): 1.61 -> 1.58 | Helped |
| Use MM-Attention instead of Add fusion (ablation) | OmniTalker · Fusion ablation | WER(%): 2.89 -> 1.58 | Helped |
| Use MM-Attention instead of Linear fusion (ablation) | OmniTalker · Fusion ablation | WER(%): 2.32 -> 1.58 | Helped |
| Use MM-Attention instead of Cross-Attention (ablation) | OmniTalker · Fusion ablation | WER(%): 1.72 -> 1.58 | Helped |
| Use MM-Attention instead of Add fusion (ablation) | OmniTalker · Fusion ablation | E-FID: 0.58 -> 0.08 | Helped |
| Use MM-Attention instead of Linear fusion (ablation) | OmniTalker · Fusion ablation | E-FID: 0.21 -> 0.08 | Helped |
| Use MM-Attention instead of Cross-Attention (ablation) | OmniTalker · Fusion ablation | E-FID: 0.12 -> 0.08 | Helped |
| Use logit-norm rather than uniform timestep sampling (ablation) | OmniTalker · Training | model performance | Helped |
Surprising findings
- Adding visual conditioning reduced WER on the Custom dataset (1.58 vs. 1.61 without), which the authors attribute to multi-task audio-visual learning.
- Sync-C is slightly lower than the diffusion baseline Hallo, which the authors attribute to the metric favoring front-facing video.
Limitations
- Limited to head region; no hand or full-body motion modeling.
- GAN-based renderer produces texture blurring and boundary artifacts at large motion amplitudes.
- Baselines' FPS excludes the TTS inference time, so real-time claim for the full cascade is not established for them.
Experimental setup
Datasets: TalkingHead-1KH VoxCeleb CelebV-HQ SEED VoxCeleb2
Benchmarks: SEED VoxCeleb2 Custom
Metrics: WER SIM-A FVD CSIM Sync-C E-FID P-FID FPS MOS
Baselines: CosyVoice MaskGCT F5-TTS SadTalker AniTalker StyleTalk EchoMimic Hallo
Builds on: flow matching MMAudio F5-TTS LivePortrait Vocos
Closest papers by meaning
- Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style (AAAI 2024 · similarity 0.90)
- MoDiTalker: Motion-Disentangled Diffusion Model for High-Fidelity Talking Head Generation (AAAI 2025 · similarity 0.90)
- EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion (CVPR 2025 · similarity 0.89)
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.88)
- AnyTalk: Multi-modal Driven Multi-domain Talking Head Generation (AAAI 2025 · similarity 0.88)
- TalkCuts: A Large-Scale Dataset for Multi-Shot Human Speech Video Generation (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.