Research map / Video / Video generation
MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MotionAura, a discrete-diffusion text-to-video and sketch-guided video inpainting framework built on a 3D masked-frame VQ-VAE (3D-MBQ-VAE) and a Fourier-based spectral transformer denoiser.
Problem
Video generation and compression need spatiotemporally consistent tokenization and efficient generative modeling; existing 3D VAEs and autoregressive or continuous-latent diffusion models struggle with temporal consistency and high-dimensional video latents.
Approach
Combines a 3D MBQ-VAE with full-frame masking and a masked-frame-index loss, a discrete (masked-token) diffusion denoiser whose attention operates in the Fourier domain with RoPE, and LoRA adaptation conditioned on sketches and text for inpainting.
Key results
- 3D-MBQ-VAE reaches 31.2 PSNR / 0.092 LPIPS on COCO-Val versus 27.7 PSNR / 0.135 LPIPS for CV-VAE(3D) at 4x compression.
- MotionAura-L reaches FVD 344 and CLIPSIM 0.2822 on MSR-VTT with WebVid-10M-recaptioned pretraining, versus 357 and 0.2429 for CogVideoX-5B.
- On sketch-guided inpainting, MotionAura-L achieves FVD 685 on DAVIS and 657 on YouTube-VOS, outperforming CogVideoX-5B (718 and 677).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MFI loss during VAE training (ablation) | 3D-MBQ-VAE · COCO-Val | LPIPS: 0.112 -> 0.092 | Helped |
| add MFI loss during VAE training (ablation) | 3D-MBQ-VAE · WebVid-Val | LPIPS: 0.134 -> 0.108 | Helped |
| jointly train on videos and images (ablation) | 3D-MBQ-VAE · COCO-Val | PSNR: 31.2 -> 33.0 | Helped |
| pretrain on WebVid 10M-recaptioned instead of WebVid 10M | MotionAura-L · MSR-VTT | FVD: 374 -> 344 | Helped |
| remove FFT and inverse FFT layers (ablation) | MotionAura-L · MSR-VTT, WebVid 10M-recaptioned | FVD: 344 -> 351 | Hurt |
| add sketch conditioning (ablation) | MotionAura-L · YouTube-VOS sketch-guided inpainting | CLIPSIM: 0.3061 -> 0.3511 | Helped |
| increase input text length from 30 to 180 tokens (ablation) | MotionAura · MSR-VTT | FVD: 364 -> 344 | Helped |
| increase LoRA rank from 8 to 64 (ablation) | MotionAura · YouTube-VOS sketch-based inpainting | CLIPSIM: 0.3291 -> 0.3511 | Helped |
| use RoPE instead of sinusoidal positional embeddings (ablation) | MotionAura · MSR-VTT, WebVid 10M | FVD: 379 -> 374 | Helped |
| use RoPE instead of sinusoidal positional embeddings (ablation) | MotionAura · MSR-VTT, WebVid 10M-recaptioned | FVD: 349 -> 344 | Helped |
| use full-frame masking during VAE training (ablation) | 3D-MBQ-VAE · WebVid-Val | LPIPS: 0.1404 -> 0.1081 | Helped |
| use random spatial masking during VAE training (ablation) | 3D-MBQ-VAE · WebVid-Val | LPIPS: 0.1334 -> 0.1081 | Helped |
Surprising findings
- Reported 3D-MBQ-VAE reconstruction on the MCL-JCV compression benchmark lags traditional codecs HEVC and VCC on PSNR and SSIM, though it has a lower LPIPS.
Limitations
- MotionAura-L is slower than the smaller variants and generates 10-second videos in 83 seconds.
- Reported compression results lag traditional codecs on PSNR and SSIM for MCL-JCV.
- The sketch-inpainting dataset is curated by CLIP-based matching of sketches, which may introduce mismatches.
Experimental setup
Datasets: YouTube-100M WebVid-10M YouTube-VOS DAVIS QuickDraw Sketchy COCO-2017 MSR-VTT JDB
Benchmarks: COCO-2017 validation WebVID validation MSR-VTT DAVIS (sketch inpainting) YouTube-VOS (sketch inpainting) MCL-JCV
Metrics: PSNR SSIM LPIPS FVD CLIPSIM Motion Quality Temporal Consistency inference time
Baselines: AnimateDiff CogVideoX-5B SimDA CV-VAE VAE-OSP TATS VQGAN VAE-SD2.1 T2V
Builds on: VQ-VAE MAGVIT-v2 VQ-Diffusion LoRA RoPE
Closest papers by meaning
- DSAQuant: Denoising-Stage-Aligned Quantization-Aware Training for Video Generation (NeurIPS 2026 · similarity 0.87)
- MotionDreamer: One-to-Many Motion Synthesis with Localized Generative Masked Transformer (ICLR 2025 · similarity 0.87)
- Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition (ICLR 2024 · similarity 0.87)
- CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer (ICLR 2025 · similarity 0.87)
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CVPR 2025 · similarity 0.87)
- Encapsulated Composition of Text-to-Image and Text-to-Video Models for High-Quality Video Synthesis (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.