Research map / 3D vision / 3D generation
LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LVSM, a transformer-based novel view synthesis model with minimal 3D inductive bias, in encoder-decoder (1D latent scene tokens) and decoder-only (single-stream) variants, trained on Objaverse and RealEstate10K.
Problem
Generalizable feed-forward novel view synthesis from sparse posed views typically relies on 3D representations (NeRF, 3DGS, triplanes) or 3D architectural biases (epipolar lines, plane sweeps), limiting flexibility, scalability, and rendering quality.
Approach
Replace handcrafted 3D representations and rendering equations with a fully transformer-based model using Plücker ray embeddings and full bidirectional self-attention; the encoder-decoder variant compresses inputs into fixed 1D latent tokens, while the decoder-only variant maps input view tokens and target ray tokens directly to target pixels.
Key results
- Decoder-only LVSM reaches 32.36 PSNR on GSO and 32.10 on ABO at 512 resolution, about 2.8-3 dB above GS-LRM.
- Decoder-only LVSM achieves 29.67 PSNR on RealEstate10K versus 28.10 for GS-LRM and 26.39 for MVSplat.
- A single-A100 decoder-only LVSM-small reaches 27.66 PSNR on RealEstate10K, 1.3 dB above MVSplat trained on one GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use decoder-only LVSM instead of GS-LRM | Novel view synthesis · ABO (Res-512) | PSNR: +3 dB | Helped |
| Use decoder-only LVSM instead of GS-LRM | Novel view synthesis · GSO (Res-512) | PSNR: 2.8 dB | Helped |
| Use decoder-only LVSM instead of GS-LRM | Novel view synthesis · ABO (Res-256) | PSNR: 3.5 dB | Helped |
| Use decoder-only LVSM instead of GS-LRM | Novel view synthesis · GSO (Res-256) | PSNR: 2.2 dB | Helped |
| Use decoder-only LVSM instead of GS-LRM | Novel view synthesis · RealEstate10K | PSNR: 1.6 dB | Helped |
| Use LVSM-small trained with 1 GPU instead of the prior best 1-GPU-trained model | Scene-level novel view synthesis · RealEstate10K | PSNR: 1.3 dB | Helped |
| Allocate more encoder-decoder layers to the decoder and fewer to the encoder (ablation) | Encoder-decoder LVSM · Object- and scene-level view synthesis | performance | Mixed |
| Increase the number of transformer layers (ablation) | Decoder-only LVSM · Object- and scene-level view synthesis | performance | Helped |
| Replace patch-based input tokenization with SRT's CNN tokenizer (ablation) | Encoder-decoder LVSM · Object- and scene-level view synthesis | performance | Hurt |
| Disable latent-token updating in the decoder (ablation) | Encoder-decoder LVSM · Object- and scene-level view synthesis | performance | Hurt |
| Predict target image patches independently without joint patch attention (ablation) | Encoder-decoder LVSM · Object- and scene-level view synthesis | performance | Hurt |
| Replace full self-attention with SRT's pure cross-attention decoder (ablation) | Encoder-decoder LVSM · Object- and scene-level view synthesis | performance | Hurt |
Surprising findings
- Deeper encoders in the encoder-decoder model hurt quality, suggesting the intermediate latent compression is a bottleneck.
- The encoder-decoder model degrades beyond 8 input views while the decoder-only model keeps improving with more views.
Limitations
- Deterministic models produce noisy, flickering artifacts in unseen regions and cannot hallucinate occluded content.
- Performance degrades when test aspect ratios or resolutions differ from the center-cropped training data.
- Decoder-only model's computation grows quadratically with input views, limiting rendering speed.
Experimental setup
Datasets: Objaverse RealEstate10K Google Scanned Objects Amazon Berkeley Objects
Benchmarks: Google Scanned Objects Amazon Berkeley Objects RealEstate10K
Metrics: PSNR SSIM LPIPS
Baselines: Triplane-LRM GS-LRM LGM pixelNeRF GPNR pixelSplat MVSplat Du et al.
Builds on: Large Reconstruction Model GS-LRM Scene Representation Transformer ViT QK-Norm
Closest papers by meaning
- MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views (NeurIPS 2024 · similarity 0.90)
- Novel View Synthesis with Pixel-Space Diffusion Models (CVPR 2025 · similarity 0.90)
- UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis (NeurIPS 2025 · similarity 0.89)
- Less Decoder is More Encoder: Geometric Representation Learning from Novel View Synthesis (NeurIPS 2026 · similarity 0.89)
- Resolving Representation Ambiguity in Feedforward Novel View Synthesis Transformer via Semantic-Spatial Decoupling (NeurIPS 2026 · similarity 0.88)
- NVComposer: Boosting Generative Novel View Synthesis with Multiple Sparse and Unposed Images (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.