Research map / Video / Video generation
What Remains in Sight? Autoregressive Video Decoding as Representation-Guided Context Rewriting
Lixuan He, Jihyeon Je, Yang You, Leonidas Guibas
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper reframes autoregressive long-horizon video decoding as representation-guided rewriting of the visible context, deciding what past evidence remains in sight rather than relying on implicit recent windows or caches.
Problem
Autoregressive long-horizon video generation typically decides the visible past implicitly through recent windows, compressed KV caches, or external memory, rather than explicitly selecting what evidence to keep.
Approach
Treats each autoregressive decoding step as explicitly rewriting the visible context guided by learned representations, so the model chooses which past content stays in sight for future prediction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ReCR context rewriting | Self-Forcing · VBench-Long 60s | Total Score: 78.19 -> 81.60 | Improved |
| add ReCR context rewriting | Causal-Forcing · VBench-Long 60s | Total Score: 77.41 -> 82.08 | Improved |
| add ReCR context rewriting | Causal-Forcing · VBench-Long 60s | Dynamic Degree: 56.82 -> 80.47 | Improved |
| add ReCR context rewriting | LongLive · VBench-Long 30s | Total Score: 81.32 -> 81.60 | No clear effect |
| add ReCR to training-free Deep Forcing | Deep Forcing · VBench-Long 60s | Total Score: 81.42 -> 82.27 | Improved |
| add ReCR context rewriting | Self-Forcing · MovieGen-128 30s drift | aesthetic drift: 5.81 -> 3.99 | Improved |
| add ReCR to Deep Forcing | Deep Forcing · MovieGen-128 30s drift | aesthetic drift: 3.56 -> 1.76 | Improved |
| add ReCR context rewriting | LongLive · MovieGen-128 30s drift | TA tail: +0.37 -> +1.05 | Hurt |
| add ReCR context rewriting | Self-Forcing · VBench-Long 60s | Subject Consistency: 96.67 -> 96.43 | Hurt |
| add ReCR context rewriting | Self-Forcing · 60s block latency | ms: 728 -> 712 | Improved |
| add ReCR context rewriting | Infinity-RoPE · 60s block latency | ms: 667 -> 710 | Hurt |
| use ReCR selection instead of contiguous recent, full, or random intermediate context (ablation) | ReCR · 60s rollouts under perturbation | quality-score drop | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 4 of 4 on most shared results.
- Causal-Forcing — Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation (ICML 2026) · better on 4 of 4 shared results
- Deep Forcing — Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compression (ICML 2026) · better on 3 of 3 shared results
- LongLive — LongLive: Real-time Interactive Long Video Generation (ICLR 2026) · better on 3 of 3 shared results
- Infinity-RoPE — Infinity-RoPE: Action-Controllable Infinite Video Generation Emerges From Autoregressive Self-Rollout (CVPR 2026) · better on 3 of 3 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- PackForcing: Short Video Training Suffices for Long Video Sampling and Long Context Inference (ECCV 2026 · similarity 0.89)
- VideoMAR: Autoregressive Video Generation with Continuous Tokens (NeurIPS 2025 · similarity 0.89)
- Learning-to-Memorize: Dynamic Context Management for Long-Horizon Autoregressive Video Generation (NeurIPS 2026 · similarity 0.89)
- End-to-End Training for Autoregressive Video Diffusion via Self-Resampling (ECCV 2026 · similarity 0.89)
- Video-Mirai: Autoregressive Video Diffusion Models Need Foresight (NeurIPS 2026 · similarity 0.89)
- Before It Fades: Reinforcing Temporal Representations at Inference Time in VideoLLMs (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
