Research map / Video / Video generation
OmniSync: Towards Universal Lip Synchronization via Diffusion Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OmniSync, a mask-free diffusion-transformer lip-sync framework with flow-matching progressive noise initialization and dynamic spatiotemporal CFG, plus the AIGC-LipSync benchmark of 615 AI-generated videos.
Problem
Lip synchronization in video relies on reference frames and masked-frame inpainting, which hurts identity consistency, pose robustness, occlusion handling and stylized/non-human content; audio is a weak conditioning signal and lip shape leaks from the source video. Existing benchmarks focus on photo
Approach
Train a DiT to map source video segments plus target audio to lip-synced target frames without masks, using timestep-dependent data sampling (pseudo-paired MEAD at high noise, unpaired YouTube at low noise), start inference from noised source frames instead of pure noise, and apply Gaussian-spatial, time-decaying classifier-free guidance focused on the mouth.
Key results
- On AIGC-LipSync, OmniSync reaches 97.40% generation success rate across all videos versus 92.20% for MuseTalk, and 87.78% on stylized characters versus 67.78% for MuseTalk.
- On HDTF, OmniSync achieves the best FID (7.855), FVD (199.627), CSIM (0.875) and LMD (7.097) among compared methods, but LatentSync has higher LSE-C (8.050 vs 7.309).
- In a 39-participant user study, OmniSync scores highest on all five criteria, e.g., Lip Sync Accuracy 3.923 vs 3.812 for LatentSync.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OmniSync instead of LatentSync | lip synchronization · HDTF | FID: 8.518 -> 7.855 | Helped |
| Use OmniSync instead of LatentSync | lip synchronization · HDTF | LSE-C: 8.050 -> 7.309 | Hurt |
| Use OmniSync instead of LatentSync | lip synchronization · AIGC-LipSync Benchmark | FID: 15.374 -> 10.681 | Helped |
| Use OmniSync instead of MuseTalk | lip synchronization · AIGC-LipSync Benchmark (all videos) | Generation Success Rate: 92.20% -> 97.40% | Helped |
| Use OmniSync instead of MuseTalk | lip synchronization · AIGC-LipSync Benchmark (stylized characters) | Generation Success Rate: 67.78% -> 87.78% | Helped |
| Add timestep-dependent sampling strategy (ablation) | OmniSync · Ablation study | CSIM: 0.727 -> 0.814 | Helped |
| Add timestep-dependent sampling strategy (ablation) | OmniSync · Ablation study | FVD: 549.768 -> 287.168 | Helped |
| Add progressive noise initialization (ablation) | OmniSync · Ablation study | FVD: 361.282 -> 287.168 | Helped |
| Replace DS-CFG with low static CFG (ablation) | OmniSync · Ablation study | LSE-C: 7.06 -> 4.16 | Hurt |
| Replace DS-CFG with high static CFG (ablation) | OmniSync · Ablation study | LSE-C: 7.06 -> 7.10 | Mixed |
| Replace DS-CFG with high static CFG (ablation) | OmniSync · Ablation study | FID: 15.710 -> 22.725 | Hurt |
| Add video description conditioning (ablation) | OmniSync · Qualitative video-description ablation | lip movement amplitude | Helped |
Surprising findings
- On HDTF, OmniSync does not win LSE-C; LatentSync keeps a higher score, which the authors attribute to its SyncNet-based loss.
Limitations
- Ablation with low static CFG reports missing FID/FVD values, and ablations are reported on a single internal setting.
- Training data includes a 400-hour proprietary YouTube dataset that is not described as released, limiting reproducibility.
- Timestep threshold and DS-CFG hyperparameters are reported with limited sensitivity analysis.
Experimental setup
Datasets: MEAD YouTube 400-hour dataset HDTF AIGC-LipSync Benchmark
Benchmarks: HDTF AIGC-LipSync Benchmark
Metrics: FID FVD CSIM NIQE BRISQUE HyperIQA LMD LSE-C Generation Success Rate user study Likert scores
Baselines: Wav2Lip VideoReTalking TalkLip IP-LAP Diff2Lip MuseTalk LatentSync EMO EchoMimic Hallo3 Sonic
Builds on: Diffusion Transformer Flow Matching Classifier-Free Guidance LatentSync
Closest papers by meaning
- DiVAS: Video and Audio Synchronization with Dynamic Frame Rates (CVPR 2024 · similarity 0.89)
- Lips Are Lying: Spotting the Temporal Inconsistency between Audio and Visual in Lip-Syncing DeepFakes (NeurIPS 2024 · similarity 0.88)
- Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metrics (CVPR 2025 · similarity 0.88)
- KeyFace: Expressive Audio-Driven Facial Animation for Long Sequences via KeyFrame Interpolation (CVPR 2025 · similarity 0.87)
- PointTalk: Audio-Driven Dynamic Lip Point Cloud for 3D Gaussian-based Talking Head Synthesis (AAAI 2025 · similarity 0.87)
- Sonic: Shifting Focus to Global Audio Perception in Portrait Animation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.