Research map / Video / Video generation
ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
ARLON pairs a decoder-only autoregressive Transformer that predicts coarse VQ-VAE visual tokens with an OpenSora-V1.2 DiT that is conditioned on them via adaptive-norm semantic injection, uncertainty sampling, and coarser latent training, to generate long videos.
Problem
Generating long text-to-video with rich motion and temporal consistency is hard for diffusion transformers, which are costly, slow, and tend to produce static or repetitive content over long horizons.
Approach
Use an AR Transformer to predict highly quantized latent VQ-VAE tokens that carry coarse spatial and long-range temporal layout, then inject them into a frozen-base DiT via gated adaptive norm modules, with coarser-latent training and uncertainty sampling to tolerate AR prediction noise.
Key results
- ARLON beats OpenSora-V1.2 on 8 of 11 VBench metrics, with dynamic degree rising from 47.2 to 52.8 and aesthetic quality from 56.2 to 61.0 in the text-to-video table.
- On 600-frame long video generation, ARLON reaches dynamic degree 50.42 vs 44.79 for OpenSora-V1.2 and overall consistency 26.55 vs 26.36.
- Inference for a 68-frame video on a single A100 drops from about 47 s (30 steps) to roughly 17-25 s including AR code generation, reported as a 47-64% relative efficiency gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ARLON instead of OpenSora-V1.2 | OpenSora-V1.2 · VBench | Dynamic Degree: 47.2 -> 52.8 | Helped |
| Use ARLON instead of OpenSora-V1.2 | OpenSora-V1.2 · VBench | Aesthetic Quality: 56.2 -> 61.0 | Helped |
| Use ARLON instead of OpenSora-V1.2 | OpenSora-V1.2 · VBench | Subject Consistency: 94.5 -> 93.4 | Hurt |
| Use ARLON instead of OpenSora-V1.2 | OpenSora-V1.2 · VBench | Background Consistency: 97.9 -> 97.1 | Hurt |
| Use ARLON instead of OpenSora-V1.2 | OpenSora-V1.2 · VBench-Long, 600 frames | Dynamic Degree: 44.79 -> 50.42 | Helped |
| Use ARLON instead of OpenSora-V1.2 | OpenSora-V1.2 · VBench-Long, 600 frames | Motion Smoothness: 98.94 -> 98.50 | Hurt |
| Use coarser DiT training tokens (4×16×16 rather than 4×8×8) (ablation) | ARLON · VBench ablation prompts | Dynamic Degree: 19.00 -> 32.00 | Helped |
| Replace Adaptive Norm with ControlNet (ablation) | ARLON semantic injection · VBench ablation prompts | Dynamic Degree: 32.00 -> 46.00 | Helped |
| Replace Adaptive Norm with MLP Adapter (ablation) | ARLON semantic injection · VBench ablation prompts | Subject Consistency: 97.40 -> 97.99 | Helped |
| Inject AR codes into the first 3 rather than 14 DiT layers (ablation) | ARLON · VBench ablation prompts | Dynamic Degree: 32.00 -> 28.00 | Hurt |
| Inject AR codes into the first 8 rather than 14 DiT layers (ablation) | ARLON · VBench ablation prompts | Dynamic Degree: 32.00 -> 31.00 | Hurt |
| Add Gaussian random noise to latent features (ablation) | ARLON · VBench ablation prompts | Dynamic Degree: 32.00 -> 34.00 | Helped |
Surprising findings
- Training the DiT on coarser latent tokens (4x16x16) than used at inference (4x8x8) improved robustness rather than hurting quality.
- ControlNet-style injection gave the highest dynamic degree but produced more distortions and weaker subject consistency than adaptive norm injection.
Limitations
- Built on OpenSora-V1.2, which caps video quality; the authors note potential swap to stronger DiTs.
- AR code sequence length becomes impractical for 2K resolution training and inference.
- Failure cases on fine hand movements such as applying makeup or eating.
Experimental setup
Datasets: OpenVid-1M ChronoMagic-ProH OpenSora-plan (Mixkit, Pexels, Pixabay) OpenVidHD-0.4M Mixkit VBench prompts
Benchmarks: VBench VBench-Long
Metrics: dynamic degree aesthetic quality imaging quality subject consistency background consistency motion smoothness temporal flickering temporal style overall consistency scene
Baselines: FreeNoise StreamingT2V OpenSora-V1.2 Kling Gen-2 Pika-V1.0 VideoCrafter-2.0 LaVie Show-1 CogVideo OpenSoraPlan-V1.1
Builds on: OpenSora-V1.2 VQ-VAE STDiT T5 VBench
Closest papers by meaning
- AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion (CVPR 2025 · similarity 0.90)
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CVPR 2025 · similarity 0.90)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.89)
- Q-ARVD: Quantizing Autoregressive Video Diffusion Models (NeurIPS 2026 · similarity 0.89)
- Ca2-VDM: Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharing (ICML 2025 · similarity 0.89)
- OSV: One Step is Enough for High-Quality Image to Video Generation (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.