Research map / Video / Video generation
Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FramePack, a progressive frame-context compression structure for next-frame(-section) video diffusion models with bounded context length, plus anti-drifting sampling orders and discrete history tokens, finetuned on HunyuanVideo.
Problem
Next-frame-prediction video diffusion models face a forgetting-drifting trade-off: encoding more history frames explodes transformer context length, while reducing drift (error accumulation) usually weakens temporal memory.
Approach
Compress history frames with frame-wise importance (time proximity or feature similarity) using geometrically decreasing patchify kernels so total context length converges to a fixed bound; combine with planned-endpoint bidirectional sampling orders and K-means discrete history representation to suppress drift.
Key results
- Inverted anti-drifting sampling with FramePack (f1k1_x_g9_f1k1f2k2f16k4_td) reaches ELO 1220 and best scores on all reported drifting metrics in the comparison table.
- Vanilla sampling with discrete history (K=256) achieves ELO 1224, competitive with the best while having a larger dynamic range.
- FramePack training reaches a batch size of about 64 on a single 8xA100-80G node with 13B HunyuanVideo at 480p using LoRA (window size 2 or 3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use inverted anti-drifting sampling instead of alternative architectures | HunyuanVideo · 512 text-to-video prompts, 30 seconds | Δ drift Clarity: 7.35% -> 2.25% | Helped |
| Use inverted anti-drifting sampling instead of vanilla sampling (ablation) | HunyuanVideo · 512 text-to-video prompts, 30 seconds | ELO: 1074 -> 1220 | Helped |
| Add discrete history to vanilla sampling (ablation) | HunyuanVideo · 512 text-to-video prompts, 30 seconds | ELO: 1074 -> 1222 | Helped |
| Add a planned endpoint to vanilla sampling (ablation) | HunyuanVideo · 512 text-to-video prompts, 30 seconds | Δ drift Clarity: 3.18% -> 2.72% | Helped |
| Change vanilla sampling to inverted sampling (ablation) | HunyuanVideo · 512 text-to-video prompts, 30 seconds | Dynamic: 92.97% -> 88.76% | Hurt |
| Generate 1 rather than 9 frames per step in vanilla sampling (ablation) | HunyuanVideo · 512 text-to-video prompts, 30 seconds | ELO: 1074 -> 1030 | Hurt |
| Generate 4 rather than 9 frames per step in vanilla sampling (ablation) | HunyuanVideo · 512 text-to-video prompts, 30 seconds | ELO: 1074 -> 1050 | Hurt |
| Change the tail handling option among delete, compress, and append (ablation) | FramePack sampling configurations · 512 text-to-video prompts, 30 seconds | ELO: 1142; 1145; 1150 | No clear effect |
| Vary packing configuration within the same sampling approach (ablation) | FramePack sampling configurations · 512 text-to-video prompts, 30 seconds | ELO | No clear effect |
| Use inverted sampling rather than vanilla sampling (ablation) | HunyuanVideo · 512 text-to-video prompts, 30 seconds | Motion: 94.08% -> 98.11% | Mixed |
Surprising findings
- Vanilla sampling achieves the highest dynamic score but low ELO, which the authors attribute to drifting artifacts rather than quality.
- The inverted anti-drifting sampling has a relatively small dynamic range despite best drift metrics.
Limitations
- Human preference evaluation uses ~100 A/B assessments per ablation, with ELO differences within ±16 treated as ties.
- Anti-drifting sampling reduces dynamic range, trading motion for stability.
- Endpoint planning can still drift within endpoint-wise segments when prompted sections are close.
Experimental setup
Datasets: LTXVideo-style dataset collection pipeline (multi-resolution, quality-filtered video)
Benchmarks: VBench VBench2
Metrics: Clarity (MUSIQ) Aesthetic (LAION predictor) Motion smoothness Dynamic degree (RAFT) Semantic (ViCLIP) Anatomy Identity (ArcFace) start-end drift contrast ELO from human A/B tests
Baselines: Repeating image-to-video Anchor frames (StreamingT2V-like) Causal attention (CausVid-like) Diffusion Forcing History guidance (HistoryGuidance-like)
Builds on: Diffusion Forcing CausVid RoPE HunyuanVideo
Closest papers by meaning
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CVPR 2025 · similarity 0.87)
- Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition (ICLR 2024 · similarity 0.87)
- Towards Precise Scaling Laws for Video Diffusion Transformers (CVPR 2025 · similarity 0.87)
- Looking Backward: Streaming Video-to-Video Translation with Feature Banks (ICLR 2025 · similarity 0.87)
- AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion (CVPR 2025 · similarity 0.87)
- VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.