Research map / 3D vision / 3D generation
StarGen: A Spatiotemporal Autoregression Framework with Video Diffusion Model for Scalable and Controllable Scene Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes StarGen, an autoregressive framework that conditions CogVideoX video diffusion on spatially adjacent images (via a large reconstruction model's rendered features) and temporally overlapping frames for long-range, pose-controlled scene generation.
Problem
Long-range consistent scene generation is hard because large reconstruction and video generation models can only process small areas per inference, and temporal-only autoregression accumulates errors and loses spatial consistency.
Approach
Condition each video clip on both temporally overlapping frames from the previous window and spatially adjacent posed images, injecting the latter as features rendered from a large reconstruction model into a pretrained video diffusion model via a ControlNet, rather than accumulating a point cloud.
Key results
- Sparse view interpolation on RealEstate-10K: PSNR 28.49, SSIM 0.894, LPIPS 0.088, comparable to ReconX (28.31) and above ViewCrafter (24.23).
- Perpetual view generation on RealEstate-10K (short range): PSNR 23.52 vs ViewCrafter 22.60; on Tanks-and-Temples PSNR 19.52 vs 17.25 without training on that dataset.
- Ablation on 100-frame perpetual generation shows removing spatial conditioning raises Rdist from 2.088 to 11.70 on RealEstate-10K, and removing temporal conditioning raises FID from 41.72 to 54.12.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use StarGen instead of ReconX | Sparse view interpolation · RealEstate-10K | PSNR: 28.31 -> 28.49 | Helped |
| Use StarGen instead of ReconX | Sparse view interpolation · ACID | PSNR: 28.84 -> 29.69 | Helped |
| Use StarGen instead of ReconX | Sparse view interpolation · RealEstate-10K and ACID | SSIM: 0.912 -> 0.894; 0.891 -> 0.876 | Mixed |
| Use StarGen instead of ReconX | Sparse view interpolation · RealEstate-10K and ACID | LPIPS: 0.088 -> 0.088; 0.101 -> 0.116 | Mixed |
| Use StarGen instead of ViewCrafter | Perpetual view generation · RealEstate-10K and Tanks-and-Temples | PSNR: 22.60 -> 23.52; 17.25 -> 19.52 | Helped |
| Use StarGen instead of ViewCrafter | Perpetual view generation · RealEstate-10K and Tanks-and-Temples | LPIPS: 0.195 -> 0.162; 0.341 -> 0.332 | Helped |
| Add spatial conditioning (ablation) | StarGen · Perpetual view generation, RealEstate-10K, 100-frame videos | Rdist: 11.70 -> 2.088 | Helped |
| Add spatial conditioning (ablation) | StarGen · Perpetual view generation, ACID, 100-frame videos | Rdist: 4.076 -> 3.542 | Helped |
| Add temporal conditioning (ablation) | StarGen · Perpetual view generation, RealEstate-10K, 100-frame videos | FID: 54.12 -> 41.72 | Helped |
| Replace the LRM with DUSt3R (ablation) | StarGen · Sparse view interpolation, RealEstate-10K, single clip | PSNR: 28.49 -> 26.94 | Hurt |
| Add spatial conditioning (ablation) | StarGen · Sparse view interpolation, RealEstate-10K, single clip | PSNR: 23.01 -> 28.49 | Helped |
| Add Depth Anything V2 depth input (ablation) | StarGen · Sparse view interpolation, RealEstate-10K, single clip | PSNR: 27.32 -> 28.49 | Helped |
Surprising findings
- Replacing the LRM with DUSt3R (as in ViewCrafter) degrades sparse view interpolation PSNR from 28.49 to 26.94 on RealEstate-10K.
Limitations
- Struggles with large loops: content in the last clip before closing a loop may mismatch the start, causing unrealistic interpolation.
- Generated video is not reconstructed into a 3D representation, limiting spatial consistency for 3D reconstruction.
- Training and evaluation at low resolution (256x256).
Experimental setup
Datasets: RealEstate-10K ACID DL3DV-10K CityGen (CityDreamer) Tanks-and-Temples
Benchmarks: RealEstate-10K ACID Tanks-and-Temples
Metrics: PSNR SSIM LPIPS FID Rdist Tdist
Baselines: pixelNeRF GPNR AttnRend MuRF pixelSplat MVSplat GS-LRM DepthSplat ReconX ViewCrafter InfNat0 LucidDreamer
Builds on: CogVideoX GS-LRM latenSplat ViewCrafter Depth Anything V2
Closest papers by meaning
- A 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale (NeurIPS 2026 · similarity 0.88)
- InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generation (NeurIPS 2025 · similarity 0.88)
- Generative Pre-trained Autoregressive Diffusion Transformer (NeurIPS 2025 · similarity 0.87)
- NAMVIS: Next-Scale Autoregressive Multi-View Image Synthesis (NeurIPS 2026 · similarity 0.87)
- VideoMAR: Autoregressive Video Generation with Continuous Tokens (NeurIPS 2025 · similarity 0.87)
- Autoregressive Video Generation without Vector Quantization (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.