Research map / Video / Video generation
AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation
Trung Kien Pham, I -Chieh Chen, Qifeng Chen, Long Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use dual-stream design with modality-specific queries and norms instead of single-stream (ablation) | AVTok · VGGSound reconstruction | rFAD: 10.26 -> 5.93 | Improved |
| use dual-stream design instead of single-stream (ablation) | AVTok · VGGSound reconstruction | rFVD: 14.87 -> 12.80 | Improved |
| add VFAL hierarchical training (ablation) | AVTok · VGGSound cJAVG | gFVD: 193.28 -> 138.80 | Improved |
| add VFAL hierarchical training (ablation) | AVTok · VGGSound reconstruction | rFAD: 9.38 -> 5.93 | Improved |
| add representation alignment loss L_rep (ablation) | AVTok · VGGSound cJAVG | gFAD: 75.09 -> 56.58 | Improved |
| add representation alignment loss L_rep (ablation) | AVTok · VGGSound reconstruction | rFAD: 8.48 -> 5.93 | Improved |
| add cross-modal AR prior loss L_prior (ablation) | AVTok · VGGSound cJAVG | gFVD: 249.47 -> 138.80 | Improved |
| add cross-modal AR prior loss L_prior (ablation) | AVTok · VGGSound reconstruction | rFVD: 10.63 -> 12.80 | Hurt |
| replace CAV-MAE with CAV-MAE Sync as alignment model (ablation) | AVTok · VGGSound reconstruction | rFAD: 6.40 -> 5.93 | Improved |
| replace HiFi-GAN with BigVGAN vocoder (ablation) | AVTok · VGGSound reconstruction | rFAD: 5.93 -> 5.72 | Improved |
| halve video tokens from 1024 to 512 (ablation) | AVTok · VGGSound reconstruction | rFAD: 5.93 -> 14.90 | Hurt |
| halve audio tokens from 128 to 64 (ablation) | AVTok · VGGSound reconstruction | rFAD: 5.93 -> 12.77 | Hurt |
Closest papers by meaning
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · similarity 0.91)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.90)
- Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows (CVPR 2025 · similarity 0.89)
- MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis (CVPR 2025 · similarity 0.89)
- Tell What You Hear From What You See - Video to Audio Generation Through Text (NeurIPS 2024 · similarity 0.89)
- Native Audio-Visual Alignment for Generation (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
