Research map / 3D vision / Gaussian splatting
MVGamba: Unify 3D Content Generation as State Space Sequence Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MVGamba, a lightweight feed-forward Gaussian reconstructor built on Mamba state space models that causally processes expanded multi-view token sequences to generate long Gaussian sequences for unified image-to-3D, text-to-3D and sparse-view 3D generation.
Problem
Existing feed-forward Gaussian large reconstruction models compromise multi-view information propagation with local/mixed attention or per-view processing, causing multi-view inconsistency and blurred textures in 3D generation from single image, sparse images, or text.
Approach
Replace quadratic Transformer reconstructors with a Mamba-based SSM that expands multi-view images via cross-scan into a 4x longer token sequence and decodes each token into a Gaussian, enabling causal cross-view self-refinement with linear complexity; adds RotNet, a Gumbel-Softmax selection over 32 canonical quaternions for rotation prediction.
Key results
- MVGamba (49M parameters) reports PSNR 24.82, LPIPS 0.08 and CLIP 0.895 on image-to-3D versus 23.76 PSNR for TripoSR, and a 2.97 dB PSNR gain over Triplane-Gaussian.
- On sparse-view reconstruction with 4 input views on GSO, MVGamba achieves PSNR 26.25 and LPIPS 0.069 versus 24.20 PSNR for LGM with 4 views.
- Ablation on the Human subset shows removing the stable training strategy drops PSNR from 27.13 to 26.69.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add patchify + convolution image encoder (ablation) | MVGamba · G-buffer Objaverse Human subset | PSNR: 25.20 -> 27.13 | Helped |
| add patchify + convolution image encoder (ablation) | MVGamba · G-buffer Objaverse Human subset | SSIM: 0.827 -> 0.925 | Helped |
| add patchify + convolution image encoder (ablation) | MVGamba · G-buffer Objaverse Human subset | LPIPS: 0.102 -> 0.057 | Helped |
| use Gaussian Decoder with RotNet instead of the ablated decoder (ablation) | MVGamba · G-buffer Objaverse Human subset | PSNR: 25.41 -> 27.13 | Helped |
| use Gaussian Decoder with RotNet instead of the ablated decoder (ablation) | MVGamba · G-buffer Objaverse Human subset | SSIM: 0.788 -> 0.925 | Helped |
| use Gaussian Decoder with RotNet instead of the ablated decoder (ablation) | MVGamba · G-buffer Objaverse Human subset | LPIPS: 0.096 -> 0.057 | Helped |
| add stable training strategy (ablation) | MVGamba · G-buffer Objaverse Human subset | PSNR: 26.69 -> 27.13 | Helped |
| add stable training strategy (ablation) | MVGamba · G-buffer Objaverse Human subset | SSIM: 0.910 -> 0.925 | Helped |
| add stable training strategy (ablation) | MVGamba · G-buffer Objaverse Human subset | LPIPS: 0.065 -> 0.057 | Helped |
| increase Gaussian sequence length by varying patch size (ablation) | MVGamba · sparse-view reconstruction | performance | Helped |
| give the side-view first instead of the front-view (ablation) | MVGamba · 3D content generation with an incorrectly estimated front-view depth | generation quality | Helped |
Surprising findings
- A non-pixel-aligned transformer Gaussian LRM with a deconvolution upsampler produced broken geometry and blurred textures, suggesting upsamplers hurt Gaussian reconstruction.
- Under simulated worst-case input noise on one view, the per-view-then-merge reconstructor produced corrupted results, while the causal SSM model was more robust.
Limitations
- Causal modeling can fail when the front view's depth is estimated incorrectly; mitigated only by manually changing the input order.
- Generation quality depends on off-the-shelf multi-view diffusion models, which have 3D inconsistencies and are limited to 256x256 resolution.
- Inference relies on a multi-stage pipeline (diffusion then reconstruction), with the reconstructor alone not addressing diffusion errors.
Experimental setup
Datasets: Objaverse GSO G-buffer Objaverse
Benchmarks: GSO Dreamfusion gallery prompts G-buffer Objaverse Human subset
Metrics: PSNR SSIM LPIPS CLIP score CLIP R-Precision inference time Chamfer Distance VIoU
Baselines: DreamGaussian Wonder3D One-2-3-45++ TripoSR TriplaneGaussian LGM SparseNeuS SparseGS
Builds on: Mamba 3D Gaussian Splatting 2D Gaussian Splatting Gamba LGM MVDream ImageDream
Closest papers by meaning
- MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views (NeurIPS 2024 · similarity 0.88)
- DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian Splat Generation (ICLR 2025 · similarity 0.87)
- IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation (ICML 2024 · similarity 0.87)
- L4GM: Large 4D Gaussian Reconstruction Model (NeurIPS 2024 · similarity 0.87)
- S2Gaussian: Sparse-View Super-Resolution 3D Gaussian Splatting (CVPR 2025 · similarity 0.87)
- Wonderland: Navigating 3D Scenes from a Single Image (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.