Research map / 3D vision / 3D generation
4D-LRM: Large Space-Time Reconstruction Model From and To Any View at Any Time
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces 4D-LRM, a transformer that regresses pixel-aligned 4D Gaussian primitives from posed, timestamped image tokens to reconstruct and render dynamic objects from any view at any time.
Problem
General 4D reconstruction of dynamic objects from sparse posed views at some timestamps, rendering arbitrary novel view-time combinations, with efficiency and generalization that optimization-based, geometry-based, and generative methods lack.
Approach
Extends large reconstruction models to space-time by predicting per-pixel 4D Gaussians (xyzt mean, 4D covariance) from patchified tokens that concatenate RGB, Plücker rays, and timestamp maps, so the model shares information across views and time in one feed-forward pass.
Key results
- On Consistent4D re-rendered, 4D-LRM-Large with GS-LRM initialization at 256 resolution reaches 32.177 PSNR on alternating canonical views, versus 21.606 for GS-LRM (All in).
- On the Consistent4D 4D generation benchmark, 4D-LRM-Free (chained with SV3D) reports 23.777 PSNR and LPIPS 0.117 versus L4GM's 0.120 LPIPS, with CLIP 0.94 matching the best baseline.
- The core 4D-LRM model reconstructs a 24-frame sequence in one forward pass in under 1.5 seconds on a single A100 GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace per-frame GS-LRM with joint spatiotemporal 4D-LRM using one random input view per frame | GS-LRM reconstruction · Objaverse4D, random input views (1 VPF) | PSNR: 18.738 -> 28.343 | Helped |
| replace per-frame GS-LRM with joint spatiotemporal 4D-LRM using one random input view per frame | GS-LRM reconstruction · Consistent4D, random input views (1 VPF) | PSNR: 17.201 -> 30.513 | Helped |
| increase model size from Base to Large (ablation) | 4D-LRM · Consistent4D, alternating canonical views, 256 resolution | PSNR: 27.443 -> 32.177 | Helped |
| initialize the Transformer with GS-LRM weights (ablation) | 4D-LRM-Large · Consistent4D, alternating canonical views, 256 resolution | PSNR: 27.860 -> 32.177 | Helped |
| initialize the Transformer with GS-LRM weights (ablation) | 4D-LRM-Large · Objaverse4D, alternating canonical views, 256 resolution | PSNR: 26.658 -> 30.094 | Helped |
| use free Gaussians (ablation) | 4D-LRM · Consistent4D, alternating canonical views, 256 resolution | PSNR: 32.177 -> 30.396 | Hurt |
| use free Gaussians (ablation) | 4D-LRM · Objaverse4D, alternating canonical views, 256 resolution | PSNR: 30.094 -> 28.838 | Hurt |
| use four canonical rather than random input views per frame | 4D-LRM · Objaverse4D, 4 VPF | PSNR: 27.940 vs. 27.839 | Hurt |
| use four canonical rather than random input views per frame | 4D-LRM · Consistent4D, 4 VPF | PSNR: 30.445 -> 30.850 | Helped |
| use the free-Gaussian generation variant rather than 4D-LRM-Large (ablation) | 4D-LRM 4D generation · Consistent4D | LPIPS: 0.138 -> 0.117 | Helped |
| enforce strict temporal alignment of Gaussian centers to input frame time (ablation) | 4D-LRM-Base · Consistent4D reconstruction | reconstruction performance | Hurt |
| use the Hexplane-style decomposed representation instead of unified space-time 4DGS (ablation) | 4D-LRM-Base · Consistent4D reconstruction | reconstruction performance | Hurt |
Surprising findings
- Under dense canonical input (4 views per frame), GS-LRM slightly outperforms 4D-LRM on Objaverse4D (28.710 vs 27.839 PSNR).
- Enforcing strict temporal alignment of Gaussians degrades performance, whereas pixel alignment improves it; the model redistributes Gaussians to unseen time intervals.
Limitations
- Cannot efficiently process hundreds of input images in one forward pass; maximum training resolution 256x256.
- Struggles with self-occlusion, fast motion, and non-linear trajectories, producing temporal ghosting artifacts.
- Not trained at scene level; relies on posed images and 3D inductive bias.
Experimental setup
Datasets: Objaverse4D Objaverse Diffusion4D filtered subset Consistent4D GSO
Benchmarks: Consistent4D Objaverse4D GSO
Metrics: PSNR SSIM LPIPS FVD CLIP
Baselines: GS-LRM Shape of Motion L4GM SV4D Consistent4D DG4D 4Diffusion Efficient4D GaussianFlow 4DGen STAG4D
Builds on: GS-LRM 4D Gaussian Splatting 3D Gaussian Splatting Shape of Motion SV3D
Closest papers by meaning
- Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting (ICLR 2024 · similarity 0.91)
- L4GM: Large 4D Gaussian Reconstruction Model (NeurIPS 2024 · similarity 0.90)
- 4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos (NeurIPS 2025 · similarity 0.90)
- Instant4D: 4D Gaussian Splatting in Minutes (NeurIPS 2025 · similarity 0.89)
- 4K4D: Real-Time 4D View Synthesis at 4K Resolution (CVPR 2024 · similarity 0.89)
- LRM: Large Reconstruction Model for Single Image to 3D (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.