Research map / Video / Video generation
From Inpainting to Editing: Unlocking Robust Mask-Free Visual Dubbing via Generative Bootstrapping
Xu He, Haoxian Zhang, Hejia Chen, Changyuan Zheng, Liyang Chen, Songlin Tang, Jiehui Huang, Xiaoqiang Liu, Pengfei Wan, Zhiyong Wu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train mask-free editor on pseudo-pairs instead of using mask-based inpainter directly | X-Dub · HDTF | Sync-C: 8.045 -> 8.562 | Improved |
| mask-free dubber instead of mask-based inpainting | X-Dub · X-DubBench | Success rate: 66.05% -> 96.36% | Improved |
| mask-free dubber instead of mask-based inpainting | X-Dub · User study MOS | Lip Sync MOS: 3.87 -> 4.50 | Improved |
| token concatenation of reference video instead of channel concatenation (ablation) | X-Dub G_free · HDTF | Sync-C: 7.49 -> 8.56 | Improved |
| add progressive multi-phase timestep sampling instead of uniform (ablation) | X-Dub G_free · HDTF | Sync-C: 3.85 -> 8.56 | Improved |
| add lip tuning phase with SyncNet loss (ablation) | X-Dub G_free · HDTF | Sync-C: 7.68 -> 8.56 | Improved |
| add texture tuning phase with identity loss (ablation) | X-Dub G_free · HDTF | CSIM: 0.847 -> 0.883 | Improved |
| add texture tuning phase (ablation) | X-Dub G_free · HDTF | Sync-C: 8.56 -> 8.56 | No clear effect |
| retrain LatentSync on same 600-hour corpus | LatentSync · X-DubBench | Success rate: 59.77% -> 59.77% | No clear effect |
Closest papers by meaning
- OmniSync: Towards Universal Lip Synchronization via Diffusion Transformers (NeurIPS 2025 · similarity 0.91)
- Identity-Preserving Video Dubbing Using Motion Warping (IJCV 2026 · similarity 0.90)
- KeyFace: Expressive Audio-Driven Facial Animation for Long Sequences via KeyFrame Interpolation (CVPR 2025 · similarity 0.87)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.87)
- Sonic: Shifting Focus to Global Audio Perception in Portrait Animation (CVPR 2025 · similarity 0.87)
- ProsodyTalker: 3D Visual Speech Animation via Prosody Decomposition (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
