Research map / Video / Video segmentation and tracking
DiVAS: Video and Audio Synchronization with Dynamic Frame Rates
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiVAS, a fully Transformer audio-video synchronization model that ingests raw audio and video at native frame/sample rates using a time-aware positional encoding, evaluated on LRS2, LRS3 and a new challenging film dataset.
Problem
Automatic detection of audio-video lip-sync offsets in professional film and broadcast content, which requires handling variable frame rates, sample rates, multiple faces, and noisy real-world scenes; prior CNN-based methods need fixed-size inputs and conversions to 25 fps.
Approach
A convolution-free two-stream Transformer (3D ViT for video, 1D Transformer for raw audio) with a time-aware sinusoidal positional encoding that uses real timestamps so the same model handles any frame rate and sample rate, trained with a contrastive loss on shifted negatives; it can operate on face crops or full frames and supports title-level drift/offset assessment.
Key results
- On LRS2, DiVAS reaches 99.4% accuracy at ±40 ms with 4.8M parameters, ~17x faster than SyncNet and ~350x faster than VocaLiST.
- On LRS3-multfps tested at 60 fps, DiVAS achieves 96.8% vs 82.2% for the base model at ±40 ms, showing frame-rate generalization.
- DiVAS-frame (full frames) reaches 93.6% accuracy at ±40 ms, ~30x less preprocessing time than face-crop DiVAS (317 vs 9.4 FPS prep).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add timestamp information to positional encoding (ablation) | DiVAS · LRS3-multfps, 60 fps | accuracy ([-40, 40] ms): 82.2 -> 96.8 | Helped |
| add timestamp information to positional encoding (ablation) | DiVAS · LRS3-multfps, up to 30.0 fps | accuracy ([-40, 40] ms): 94.0 -> 94.4 | Helped |
| use full video frames instead of face crops (ablation) | DiVAS · LRS3-multfps | accuracy ([-40, 40] ms): 94.4 -> 93.6 | Hurt |
| train on LRS3-multfps instead of LRS2 (ablation) | DiVAS · New dataset, clip level | accuracy ([-125, 45] ms): 73.9 -> 81.9 | Helped |
| train on the new challenging dataset instead of LRS3-multfps (ablation) | DiVAS · New dataset, clip level | accuracy ([-40, 40] ms): 61.9 -> 89.4 | Helped |
| aggregate predictions at title level instead of clip level (ablation) | DiVAS · New dataset, accuracy ([-40, 40] ms) | accuracy: 89.4 -> 93.5 | Helped |
| use full frames instead of face crops (ablation) | DiVAS · LRS3-multfps, preprocessing speed | Prep (FPS): 9.4 -> 317 | Helped |
Surprising findings
- On LRS2 tested with random offsets, SyncNet outperformed more recent approaches like VocaLiST.
- Models trained on standard benchmarks (LRS2, LRS3) generalize poorly to complex film content, with accuracy at ±40 ms dropping to 54.5-61.9%.
Limitations
- Trained with artificial offsets only; real production sync errors may differ.
- Performance on complex data is lower at strict tolerances when trained on standard benchmarks.
- Full-frame variant is slightly less accurate than face-crop variant and needs longer training to focus on the mouth.
Experimental setup
Datasets: LRS2 LRS3-multfps new challenging film dataset (218k dialog scenes, ~140 h, 170 titles)
Benchmarks: LRS2 LRS3-multfps new challenging dataset (title-level and clip-level)
Metrics: accuracy at temporal tolerances (ms) F1 score for active speaker detection FPS parameter count
Baselines: SyncNet VocaLiST Iashin et al. (AVST/full-frame sparse selector model) Base model with regular positional encoding
Builds on: SyncNet VATT VocaLiST Perfect Match
Closest papers by meaning
- OmniSync: Towards Universal Lip Synchronization via Diffusion Transformers (NeurIPS 2025 · similarity 0.89)
- Audio-Sync Video Generation with Multi-Stream Temporal Control (NeurIPS 2025 · similarity 0.86)
- MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis (CVPR 2025 · similarity 0.86)
- Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metrics (CVPR 2025 · similarity 0.86)
- Long-Term TalkingFace Generation via Motion-Prior Conditional Diffusion Model (ICML 2025 · similarity 0.85)
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.