Research map / Video / Video generation
VideoMAR: Autoregressive Video Generation with Continuous Tokens
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoMAR, a decoder-only masked autoregressive image-to-video model over continuous VAE tokens that generates frame-by-frame with spatial masked generation, using a next-frame diffusion loss, curriculum training, and progressive temperature.
Problem
Autoregressive video generation with masked generation in continuous space is under-explored; existing mask-based video AR methods either break frame-by-frame causality, cause train-inference gaps, or double sequence length, and long-sequence AR video training is costly.
Approach
Combine frame-wise causal attention (temporal causality) with per-frame masked diffusion-loss prediction (spatial bidirectionality) via a next-frame diffusion loss that keeps full previous context, plus short-to-long curriculum, progressive resolution, progressive temperature at inference, and 3D-RoPE for extrapolation.
Key results
- VideoMAR-stage2 (1.4B, 0.5M data) scores 84.82 total on VBench-I2V, above Cosmos-13B (84.22) and Cosmos-5B (84.16).
- Ablation on VBench-I2V stage 1: total score rises from 78.81 (none) to 82.56 with next-frame loss, causal attention, and temperature.
- Inference with KV cache reduces time for stage2 from 1941s (NTP) to 134s (VideoMAR), and steps from 1440 to 64 per frame.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add next-frame loss (ablation) | VideoMAR stage 1, without causal attention or temperature strategy · VBench-I2V | Total Score: 78.81 -> 79.76 | Helped |
| add causal attention (ablation) | VideoMAR stage 1 with next-frame loss · VBench-I2V | Total Score: 79.76 -> 80.72 | Helped |
| add temperature strategy (ablation) | VideoMAR stage 1 with next-frame loss and causal attention · VBench-I2V | Total Score: 80.72 -> 82.56 | Helped |
| use temperature 0.98 instead of 1.00 (ablation) | VideoMAR · VBench-I2V | Total Score: 82.19 -> 83.13 | Helped |
| use temperature 0.96 instead of 0.98 (ablation) | VideoMAR · VBench-I2V | Total Score: 83.13 -> 83.56 | Helped |
| use temperature 0.94 instead of 0.96 (ablation) | VideoMAR · VBench-I2V | Total Score: 83.56 -> 84.02 | Helped |
| use temperature 0.92 instead of 0.94 (ablation) | VideoMAR · VBench-I2V | Total Score: 84.02 -> 84.31 | Helped |
| use constant temperature 0.90 instead of 0.92 (ablation) | VideoMAR · VBench-I2V | Total Score: 84.31 -> 84.82 | Helped |
| use constant temperature 0.90 rather than progressive temperature (ablation) | VideoMAR · VBench-I2V | Dynamic Degree: 9.87; 10.98 | Hurt |
| add KV cache (ablation) | VideoMAR stage 1 · Inference | Inference time (s): 417 -> 107 | Helped |
| add KV cache (ablation) | VideoMAR stage 2 · Inference | Inference time (s): 671 -> 134 | Helped |
| use VideoMAR-stage2 instead of Cosmos 13B | Image-to-video generation · VBench-I2V | Total Score: 84.22 -> 84.82 | Helped |
Surprising findings
- Lower temperature (about 0.9) is reported to suppress accumulated error, and reducing temperature trades off dynamic degree (Dyna. Degr. drops from 24.80 at 1.00 to 9.87 at 0.90).
Limitations
- Trained from scratch on only 0.5M internal videos, so it lags on complex/large motions and on latest diffusion models (Magi-1, Step-Video-TI2V).
- Only image-to-video and video-to-video explored; text-to-video, editing and interactive world modeling left for future work.
- Dynamic degree is lower than competing methods, and reducing temperature lowers dynamic degree.
Experimental setup
Datasets: 0.5M internal video-text pairs
Benchmarks: VBench-I2V
Metrics: Total Score I2V Score Quality Score I2V Subject I2V Background Camera Motion Subject Consistency Background Consistency Motion Smoothness Dynamic Degree
Baselines: Cosmos-5B Cosmos-13B Magi-1 Step-Video-TI2V CogVideoX-I2V SEINE I2VGen-XL ConsistI2V VideoCrafter-I2V NOVA MAGI
Builds on: MAR Cosmos-Tokenizer VBench-I2V RoPE FAR
Closest papers by meaning
- Autoregressive Video Generation without Vector Quantization (ICLR 2025 · similarity 0.90)
- CanvasMAR: Improving Masked Autoregressive Video Prediction With Canvas (NeurIPS 2026 · similarity 0.90)
- InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generation (NeurIPS 2025 · similarity 0.90)
- Parallelized Autoregressive Visual Generation (CVPR 2025 · similarity 0.89)
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CVPR 2025 · similarity 0.89)
- AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.