Research map / Video / Video generation
Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Divot, a video tokenizer trained by using a video diffusion model to denoise VAE latents conditioned on tokenizer features, and Divot-LLM, which models those continuous features with a GMM head for video comprehension and generation.
Problem
Unifying video comprehension and generation within an LLM requires a video tokenizer that yields continuous spatiotemporal representations usable for understanding and decodable into realistic videos; prior discrete tokenizers degrade comprehension.
Approach
Use a video diffusion model as the self-supervised proxy objective for training a continuous video tokenizer (the diffusion model conditioned on tokenizer features denoises VAE latents, and doubles as a de-tokenizer), and model the tokenizer's continuous features inside an LLM with a Gaussian Mixture Model trained by NLL rather than MSE or diffusion heads.
Key results
- Divot-LLM (7B) reaches 46.5 on EgoSchema, 58.3 on Perception-Test, 52.1 on MVBench, 76.4 on MSVD and 55.8 on ActivityNet, outperforming VILA-U and Video-LaVIT on MSVD and ActivityNet among unified video models.
- Zero-shot text-to-video on MSR-VTT gives CLIPSIM 0.2938 and FVD 301.4 using 4.8M video-text pairs, comparable to VILA-U (0.2937, 499.06) and Video-LaVIT (0.3012, 188.36).
- In the generation ablation, GMM modeling achieves CLIPSIM 0.3265 and FVD 366.60, better than MSE regression (0.2386 CLIPSIM) and diffusion modeling (0.3192 / 378.50 ε-pred).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use diffusion loss rather than captioning loss to train the video tokenizer (ablation) | Divot tokenizer · MV-Bench | accuracy: 30.8 -> 33.2 | Helped |
| Use diffusion loss rather than captioning loss to train the video tokenizer (ablation) | Divot tokenizer · MSVD | accuracy: 66.1 -> 68.9 | Helped |
| Use diffusion loss rather than captioning loss to train the video tokenizer (ablation) | Divot tokenizer · ActivityNet | accuracy: 43.2 -> 44.3 | Helped |
| Use patch-position independent Perceiver Resampler features instead of patch-position dependent features (ablation) | Divot-LLM video generation · MSR-VTT | CLIPSIM: 0.3192 -> 0.3265 | Helped |
| Use patch-position independent Perceiver Resampler features instead of patch-position dependent features (ablation) | Divot-LLM video generation · MSR-VTT | FVD: 378.50 -> 366.60 | Helped |
| Use GMM modeling instead of MSE regression for video features (ablation) | Divot-LLM video generation · MSR-VTT | CLIPSIM: 0.3168 -> 0.3265 | Helped |
| Use GMM modeling instead of MSE regression for video features (ablation) | Divot-LLM video generation · MSR-VTT | FVD: 438.94 -> 366.60 | Helped |
| Use GMM modeling instead of diffusion modeling with ϵ prediction (ablation) | Divot-LLM video generation · MSR-VTT | CLIPSIM: 0.2811 -> 0.3265 | Helped |
| Use GMM modeling instead of diffusion modeling with v prediction (ablation) | Divot-LLM video generation · MSR-VTT | FVD: 377.17 -> 366.60 | Helped |
| Use query-based prediction rather than autoregressive prediction for GMM modeling (ablation) | Divot-LLM video generation · MSR-VTT | CLIPSIM: 0.2386 -> 0.3265 | Helped |
| Use query-based prediction rather than autoregressive prediction for GMM modeling (ablation) | Divot-LLM video generation · MSR-VTT | FVD: 447.88 -> 366.60 | Helped |
| Use bidirectional rather than causal attention for query-based GMM modeling (ablation) | Divot-LLM video generation · MSR-VTT | CLIPSIM: 0.3080 -> 0.3265 | Helped |
Surprising findings
- Simple MSE regression of LLM outputs to video features yields repeating patterns and the worst generation quality, and diffusion heads on the LLM output also underperform GMM modeling.
- Autoregressive token-by-token feature prediction performs worst, attributed to error accumulation when sampling from the GMM.
Limitations
- Model predicts video representations for only a single short clip and cannot generate longer videos.
- Uses only a 7B Mistral LLM and a relatively small pre-training set, so comparisons with larger video LLMs are not on equal footing.
- Zero-shot generation FVD and CLIPSIM are only moderately competitive with specialized generators.
Experimental setup
Datasets: WebVid-10M Panda-70M CC3M CapsFusion LAION-COCO Kinetics-710 SSV2 TGIF NextQA CLEVRER YouCook2 PerceptionTest
Benchmarks: EgoSchema Perception-Test MVBench MSVD ActivityNet MSR-VTT
Metrics: accuracy CLIPSIM FVD
Baselines: Video-LaVIT VILA-U LWM Loong VideoLLaMA 2 LLaVA-NeXT-Video VideoChat2 CogVideo VideoPoet
Builds on: DynamiCrafter MAR GIVT Mistral-7B LoRA
Closest papers by meaning
- Mimir: Improving Video Diffusion Models for Precise Text Understanding (CVPR 2025 · similarity 0.90)
- Language Model Beats Diffusion - Tokenizer is key to visual generation (ICLR 2024 · similarity 0.90)
- LLM-grounded Video Diffusion Models (ICLR 2024 · similarity 0.89)
- Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition (ICLR 2024 · similarity 0.88)
- VideoTetris: Towards Compositional Text-to-Video Generation (NeurIPS 2024 · similarity 0.88)
- TokenFlow: Consistent Diffusion Features for Consistent Video Editing (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.