Research map / Video / Video generation
Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes diffusion latent beam search with a lookahead reward estimator plus a calibrated linear combination of video quality metrics to improve text-to-video generation at inference time without weight updates.
Problem
Text-to-video diffusion outputs often show unnatural motion, deformation, reverse playback, or motionless scenes; existing inference-time alignment (greedy search, best-of-N, SMC) is noisy and existing single video metrics poorly match VLM/human judgments.
Approach
Beam search over diffusion latents (K candidates per beam, top-B kept) scored by a lookahead estimator that runs deterministic DDIM steps to reduce reward-estimate noise, combined with a reward calibrated by brute-force weighting of six perceptual metrics to maximize Pearson correlation with Gemini scores.
Key results
- DLBS improves calibrated reward as search budget KB increases (up to 32), while BoN and greedy search saturate in some cases; DLBS-LA with KB=8, T'=6 is comparable to or better than DLBS with KB=32.
- Human pairwise comparisons by three raters show DLBS-LA (KB=8, NFE=2500) consistently preferred over BoN (KB=64, NFE=3200) across models and prompts.
- Calibrated reward achieves the highest Pearson correlation with Gemini across all prompt sets (statistically significant, p<0.01), outperforming single metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine perceptual video metrics with calibrated weights instead of optimizing a single metric | Reward calibration · DEVIL prompt sets and MSRVTT-test | Pearson correlation with Gemini feedback | Helped |
| Increase the DLBS search budget | DLBS · DEVIL prompt sets and MSRVTT-test | Gemini-calibrated reward | Helped |
| Use DLBS instead of best-of-N sampling or greedy search | Inference-time text-to-video alignment · DEVIL prompt sets and MSRVTT-test | Gemini-calibrated reward at matched compute | Helped |
| Add the lookahead estimator to DLBS | DLBS · DEVIL prompt sets and MSRVTT-test | Calibrated reward | Helped |
| Use DLBS-LA rather than BoN | Latte, CogVideoX, and Wan 2.1 · Human pairwise evaluation | Human preference | Helped |
| Use DLBS to optimize calibrated rewards | Latte · DEVIL prompt sets and MSRVTT-test | Gemini or GPT-4o feedback score | Helped |
| Use DLBS to optimize calibrated rewards | Latte · DEVIL prompt sets and MSRVTT-test | VideoScore | Helped |
| Apply DLBS-LA instead of greedy search | Latte · Movie Gen Video Bench | Alignment reward | Helped |
| Add DLBS after VideoDPO fine-tuning | VideoCrafter2 with VideoDPO · DEVIL-high and MSRVTT-test | Performance | Helped |
| Apply VideoDPO fine-tuning without DLBS | VideoCrafter2 · DEVIL-high and MSRVTT-test | Performance | No clear effect |
| Increase the lookahead horizon from the posterior-mean estimate to more lookahead steps (ablation) | DLBS-LA · MSRVTT-test | Gemini reward | Helped |
| Increase lookahead steps beyond the modest tested horizons (ablation) | DLBS-LA · MSRVTT-test | Search performance | No clear effect |
Surprising findings
- Increasing diffusion steps T yields only marginal gains compared with spending compute on lookahead and search budget.
- A single metric such as subject consistency or dynamic degree often correlates poorly with VLM judgments, and optimizing a single metric leads to over-optimization.
Limitations
- Relies on VLMs (Gemini, GPT-4o) as proxies for human evaluation; reward calibration depends on them.
- Experiments limited to 17-33 frames per sample for CogVideoX and Wan 2.1 due to compute constraints.
- Search costs many reward evaluations, requiring cheap proxy rewards.
Experimental setup
Datasets: DEVIL MSRVTT Movie Gen Video Bench
Benchmarks: DEVIL MSRVTT-test Movie Gen Video Bench
Metrics: calibrated reward Gemini-1.5 score (1-10) GPT-4o score (1-10) VideoScore human pairwise preference mean pairwise distance of ViCLIP embeddings (diversity) Pearson correlation
Baselines: best-of-N sampling greedy search posterior mean reward estimate VideoDPO
Builds on: DDIM classifier guidance twisted SMC VBench
Closest papers by meaning
- MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance (AAAI 2024 · similarity 0.90)
- VideoDPO: Omni-Preference Alignment for Video Diffusion Generation (CVPR 2025 · similarity 0.89)
- VideoElevator: Elevating Video Generation Quality with Versatile Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.89)
- Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution (CVPR 2024 · similarity 0.89)
- TI2V-Zero: Zero-Shot Image Conditioning for Text-to-Video Diffusion Models (CVPR 2024 · similarity 0.89)
- TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.