Research map / Video / Video generation
Fused View-Time Attention and Feedforward Reconstruction for 4D Scene Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a two-stage 4D scene generator: a fused view-time masked-attention video diffusion model (11B DiT, no new parameters) producing synchronized multi-view video grids, followed by a feedforward VGGT-based reconstruction model with a Gaussian head, camera token replacement, and temporal attention that outputs dynamic 3D Gaussians.
Problem
Generating 4D scenes (synchronized multi-view, time-varying video) with high visual quality and consistency, and converting the generated 4D video grids into renderable Gaussian splats efficiently without per-scene optimization or known camera poses.
Approach
Fuse cross-view and cross-time attention into a single masked self-attention layer (each token attends to tokens sharing its view or timestamp), implemented via FlexAttention sparsity, so no new parameters are added to the pretrained video DiT; pair it with a feedforward VGGT-based reconstructor that predicts cameras and dynamic Gaussians jointly.
Key results
- Fused view-time attention reaches 22.49 PSNR / 0.909 SSIM / 0.113 LPIPS on Objaverse vs 21.40 PSNR for parallel architecture and 5.94 for sequential.
- On Neural3DVideo without manual scale tuning, the reconstruction model achieves 21.63 PSNR / 0.1200 LPIPS, above BTimer multi-view (21.55 with manual scale tuning).
- On the generated video dataset, the full method reaches Met3R 0.173 vs 0.192 for 4Real-Video and 0.530 for SynCamMaster (lower is better).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use fused view-time attention instead of parallel attention | 4D video model · Objaverse | PSNR: 21.40 -> 22.49 | Helped |
| use fused view-time attention instead of sequential attention | 4D video model · NVIDIA Dynamic Dataset | LPIPS: 0.147 -> 0.142 | Helped |
| remove 2D transformed videos from training (ablation) | 4D video model · Generated video dataset | Image: 66.2 -> 63.9 | Hurt |
| remove camera token replacement (ablation) | feedforward reconstruction model · Dynamic Kubric test set | PSNR: 23.39 -> 22.48 | Hurt |
| remove temporal attention (ablation) | feedforward reconstruction model · Dynamic Kubric test set | PSNR: 23.39 -> 22.60 | Hurt |
| use 4 diffusion steps instead of 40 | 4D video model · Generated 4D videos | Met3R: 0.173 -> 0.187 | Hurt |
| use classifier-free guidance when both reference video types are provided | 4D video model · Generated 4D videos | output quality | No clear effect |
| use the (v+t,x,y) positional embedding mapping instead of the proposed mapping | 4D video model · Generation conditioned on only one reference video type | output stability | Hurt |
Surprising findings
- Parallel architecture with synchronization layers underperforms the fused design, suggesting bottleneck layers limit capacity.
- Removing 2D transformed videos reduces accuracy only modestly on Met3R but the paper notes it is a key augmentation for limited real 4D data.
Limitations
- Does not support full 360-degree scene generation.
- Layering artifacts can appear in reconstructed Gaussian splats.
- Inference is computationally expensive (~4 minutes for 8 views and 29 frames on a single A100).
Experimental setup
Datasets: Objaverse Kubric RealEstate10K DL3DV MVImageNet ACID Dynamic Objaverse NVIDIA Dynamic Dataset Neural3DVideo Tanks & Temples LLFF
Benchmarks: Objaverse NVIDIA Dynamic Dataset Generated video dataset (Veo 2) Tanks & Temples LLFF Neural3DVideo Kubric test
Metrics: PSNR SSIM LPIPS Met3R VBench flickering VBench motion VBench subject VBench background VBench image quality
Baselines: SV4D TrajectoryCrafter ReCamMaster SynCamMaster 4Real-Video GSLRM BTimer Splatt3r
Builds on: VGGT ReCamMaster 4Real-Video Splatt3r FlexAttention
Closest papers by meaning
- 4Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusion (CVPR 2025 · similarity 0.89)
- CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models (CVPR 2025 · similarity 0.89)
- Instant4D: 4D Gaussian Splatting in Minutes (NeurIPS 2025 · similarity 0.89)
- 4Diffusion: Multi-view Video Diffusion Model for 4D Generation (NeurIPS 2024 · similarity 0.89)
- EG4D: Explicit Generation of 4D Object without Score Distillation (ICLR 2025 · similarity 0.89)
- 4D-LRM: Large Space-Time Reconstruction Model From and To Any View at Any Time (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.