Research map / Video / Video generation
Align-A-Video: Deterministic Reward Tuning of Image Diffusion Models for Consistent Video Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Align-A-Video tunes an image diffusion model with an HPSv2 reward on a single anchor frame using deterministic (fixed-noise) LoRA reward tuning, then propagates the optimized features to the whole video via cross-frame attention for zero-shot-style video editing.
Problem
Zero-shot video editing with image diffusion models preserves temporal consistency via strong structural control but often fails to follow edit prompts and yields lower visual quality; directly applying image reward tuning to videos harms temporal consistency and optimization is unstable.
Approach
Stabilize single-prompt reward fine-tuning by fixing the DDIM-inverted initial noise, sharing inversion features for the first k denoising steps, and randomly truncating gradients; optimize only one anchor frame and propagate its features to the other frames through cross-frame attention and distance-weighted fusion.
Key results
- On V2VBench (50 videos), the method reaches DOVER 0.761 versus 0.708 for the second-best (ControlVideo), the best visual quality among compared methods.
- Pick score 21.847 and CLIP score 0.955-range temporal consistency is competitive, with the method achieving the top Pick score (21.847) versus TokenFlow's 20.594.
- Reward tuning completes in minutes and reduces GPU memory, with runtime and memory compared on a single RTX 6000 Ada for a 40-frame video.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use reward fine-tuning | Align-A-Video · V2VBench | Pick score: 21.039 -> 21.847 | Helped |
| Use reward fine-tuning | Align-A-Video · V2VBench | DOVER score: 0.708 -> 0.761 | Helped |
| Use reward fine-tuning | Align-A-Video · V2VBench | CLIP score: 0.282 -> 0.284 | Helped |
| Use reward fine-tuning | Align-A-Video · V2VBench | ViCLIP score: 0.263 -> 0.265 | Helped |
| Use reward fine-tuning | Align-A-Video · V2VBench | CLIP consistency: 0.957 -> 0.955 | Hurt |
| Use reward fine-tuning | Align-A-Video · V2VBench | DINO consistency: 0.956 -> 0.944 | Hurt |
| Use reward fine-tuning | Align-A-Video · V2VBench | EPE: -1.762 -> -1.874 | Hurt |
| Add deterministic reward tuning (ablation) | Align-A-Video · V2VBench video editing | text alignment | Helped |
| Replace cross-frame attention with extended attention (ablation) | Align-A-Video · V2VBench video editing | visual quality | Hurt |
| Use PickScore or the LAION aesthetic predictor as the reward model instead of HPSv2 (ablation) | Align-A-Video · V2VBench video editing | visual quality | Helped |
Surprising findings
- Feature-injection methods (TokenFlow, Flatten) lead on temporal consistency metrics such as EPE, and the method does not win every metric.
Limitations
- Temporal consistency is not the best on all metrics; feature-injection baselines win on EPE and some consistency scores.
- Evaluation limited to 50 videos from V2VBench with image-level reward models rather than a video reward.
Experimental setup
Datasets: V2VBench
Benchmarks: V2VBench
Metrics: Aesthetic score Pick score DOVER score CLIP score ViCLIP score CLIP consistency DINO consistency EPE
Baselines: TokenFlow Flatten Tune-A-Video ControlVideo Text2Video-Zero
Builds on: DDIM inversion PnP diffusion TokenFlow AlignProp Tune-A-Video LoRA
Closest papers by meaning
- InstructVideo: Instructing Video Diffusion Models with Human Feedback (CVPR 2024 · similarity 0.90)
- FADE: Frequency-Aware Diffusion Model Factorization for Video Editing (CVPR 2025 · similarity 0.88)
- Transforming Image Editors into Video Editors (NeurIPS 2026 · similarity 0.88)
- Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search (NeurIPS 2025 · similarity 0.88)
- Improving Video Generation with Human Feedback (NeurIPS 2025 · similarity 0.87)
- Ouroboros-Diffusion: Exploring Consistent Content Generation in Tuning-free Long Video Diffusion (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.