Research map / 3D vision / Gaussian splatting
L4GM: Large 4D Gaussian Reconstruction Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes L4GM, a feed-forward 4D large reconstruction model that extends the pretrained LGM with temporal self-attention to turn single-view video into per-frame 3D Gaussians, trained on a 12M-video Objaverse-4D multiview dataset.
Problem
Generating animated 3D objects (4D content) from monocular video is ill-posed and existing methods rely on slow per-scene optimization or score distillation taking minutes to hours, while 4D training data is scarce.
Approach
Extend a pretrained 3D Large Gaussian Reconstruction Model with temporal self-attention layers so it reconstructs a Gaussian set per frame from a monocular video plus a single set of generated multiview images of the first frame, trained on a large rendered multiview animated-object dataset; add a 4D interpolation model fine-tuned from it for higher frame rates.
Key results
- On the Consistent4D benchmark L4GM reaches LPIPS 0.12, CLIP 0.94, FVD 691.87 vs. best baseline STAG4D (0.13, 0.91, 992.21), in about 3 s per video.
- User study (24 examples, 5 raters each) favors L4GM over DG4D, OpenLRM, and STAG4D on all criteria, e.g. overall quality 65.4% vs 25.0% against DG4D.
- Without LGM 3D pretraining the model fails to converge; a frozen-LGM variant and a HexPlane deformation variant converge to lower PSNR and the HexPlane version stays static.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize from pretrained LGM rather than training without 3D pretraining (ablation) | L4GM · Objaverse-4D reconstruction | PSNR | Helped |
| Fine-tune the non-temporal LGM layers instead of freezing them (ablation) | L4GM · Objaverse-4D reconstruction | PSNR | Helped |
| Add temporal self-attention (ablation) | L4GM · Objaverse-4D reconstruction | PSNR | Helped |
| Use per-frame 3D Gaussians rather than a canonical representation with a HexPlane deformation field (ablation) | L4GM · Objaverse-4D reconstruction | PSNR | Helped |
| Add sinusoidal time embeddings to the camera embedding (ablation) | L4GM · Objaverse-4D reconstruction | PSNR | No clear effect |
| Use longer input video length T when ground-truth multiviews are provided (ablation) | L4GM · Objaverse-4D reconstruction | per-frame reconstruction PSNR | Hurt |
| Increase the number of autoregressive reconstruction runs (ablation) | L4GM · Objaverse-4D reconstruction | per-frame reconstruction PSNR | Hurt |
| Apply the 4D interpolation model (ablation) | L4GM · Consistent4D videos | framerate | Helped |
Surprising findings
- A HexPlane-based deformation-field design failed to learn meaningful motion at scale, with outputs remaining static, unlike optimization-based 4D generation.
- Without 3D pretraining from LGM, the model failed to converge at all on the 4D dataset.
Limitations
- Cannot handle motion ambiguity; unnatural leg motion from non-input views in some walking motions.
- Struggles with multiple occluding objects.
- Fails on egocentric viewpoints since it assumes 0 degree elevation input views.
Experimental setup
Datasets: Objaverse-4D Objaverse 1.0 GObjaverse Consistent4D ActivityNet
Benchmarks: Consistent4D
Metrics: LPIPS CLIP image similarity FVD runtime user preference
Baselines: Consistent4D 4DGen GaussianFlow STAG4D DreamGaussian4D Efficient4D OpenLRM
Builds on: LGM LRM ImageDream 3D Gaussian Splatting Objaverse
Closest papers by meaning
- 4D-LRM: Large Space-Time Reconstruction Model From and To Any View at Any Time (NeurIPS 2025 · similarity 0.90)
- Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models (CVPR 2024 · similarity 0.89)
- Robust Multi-Object 4D Generation for In-the-wild Videos (CVPR 2025 · similarity 0.88)
- World from Motion: Generative Dynamic Gaussian Reconstruction from Monocular Video (NeurIPS 2026 · similarity 0.88)
- Instant4D: 4D Gaussian Splatting in Minutes (NeurIPS 2025 · similarity 0.88)
- LRM-Zero: Training Large Reconstruction Models with Synthesized Data (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.