Research map / Video / Video generation

Attend Locally, Remember Linearly: Linear Attention as Cross-Frame Memory for Autoregressive Video Diffusion

NeurIPS 2026 · Video › Video generation · approach: Efficient video diffusion models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Converts a pretrained autoregressive video diffusion model (Causal Forcing / Wan2.1-1.3B) into a hybrid architecture that keeps intra-frame softmax attention and replaces the inter-frame KV cache with a fixed-size gated delta network state, via two-stage distillation.

Problem

Autoregressive video diffusion relies on softmax self-attention with a linearly growing KV cache, giving quadratic compute and linear memory in video length, which limits streaming and long-horizon generation.

Approach

Decompose self-attention into an intra-frame softmax branch and an inter-frame gated delta network recurrent branch with a fixed-size state; share a pre-update state across all tokens in a frame and update the state only after the clean denoised pass; distill a pretrained AR model layer-wise with sensitivity-guided layer selection, then jointly.

Key results

• ARL2 (75% layers replaced) reaches 2.26x speedup and 54% peak memory reduction at 921 frames (40.5 GB vs 87.9 GB for Causal Forcing).
• ARL2 (50%) achieves VBench Quality Avg 87.17, the highest among distilled AR models, vs 87.09 for Causal Forcing, with better temporal flickering (97.26 vs 96.27).
• At 1005 frames Causal Forcing runs out of memory (>91 GB) while ARL2 completes using 43 GB.

Surprising findings

• Frame-level state access outperforms token-level sequential state updates on quality, and elementwise gating performs worse than headwise gating due to poor convergence.
• At 75% replacement the quality drops notably on long videos (501 and 921 frames: Quality Avg 77.00 and 77.34), while the semantic score for the 75% setting is lower than for 50% at some lengths.

Limitations

• Fixed-size state gives weaker spatial relationship and human action performance, per the authors.
• Quality degrades at 75% replacement on long videos (77.34 at 921 frames vs 83.37 for Self Forcing).
• Semantic scores at 75% replacement are lower than the teacher on several lengths.

Experimental setup

Datasets: VidProM VBench prompts

Benchmarks: VBench VBench per-dimension (subject/background consistency, temporal flickering, motion smoothness, aesthetic, imaging quality

Metrics: VBench Quality Avg VBench Semantic Avg VBench Total generation time (s) FPS peak GPU memory (GB) speedup

Baselines: Causal Forcing Self Forcing Wan2.1-T2V LTX-Video SANA-Video SkyReels-V2 MAGI-1

Builds on: Causal Forcing Gated Delta Network Self Forcing Diffusion Forcing

Closest papers by meaning

• VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion (NeurIPS 2026 · similarity 0.92)
• AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion (CVPR 2025 · similarity 0.89)
• Faster Video Diffusion with Trainable Sparse Attention (NeurIPS 2025 · similarity 0.89)
• From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CVPR 2025 · similarity 0.89)
• Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (NeurIPS 2025 · similarity 0.89)
• Bidirectional Sparse Attention for Faster Video Diffusion Training (NeurIPS 2026 · similarity 0.88)