Research map / 3D vision / 3D generation
SAR3D: Autoregressive 3D Object Generation and Understanding via Multi-scale 3D VQVAE
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAR3D, a multi-scale 3D VQVAE over latent triplanes enabling next-scale autoregressive 3D object generation (0.82 s on A6000) and an LLM (Vicuna-7B) fine-tuned on truncated 3D tokens for 3D captioning.
Problem
Autoregressive modeling has been underexplored for large-scale 3D object generation and detailed 3D understanding; existing 3D generators are slow diffusion models and 3D LLMs use coarse point clouds.
Approach
Adapts VAR-style next-scale prediction to a multi-scale latent triplane VQVAE that encodes multiview RGB-D and Plücker renderings, and reuses truncated scale tokens from the same tokenizer as LLM input for 3D captioning.
Key results
- SAR3D-NeRF achieves FID 22.55 and COV 74.17% in image-conditioned generation, better than LGM (FID 19.93 but worse MUSIQ and 3D metrics) and other baselines on the reported metrics among 3D-quality measures.
- SAR3D generates a 3D object in 0.82 s on a single A6000 GPU (1.64 s for the NeRF variant as reported in Table 1 scaling).
- Truncated scale tokens (r1..rK-2) using 37.5% of the generation tokens suffice for SAR3D-LLM to caption 3D models with category and fine details.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SAR3D-NeRF instead of the strongest listed baseline, LGM (V=4) | Image-conditioned 3D generation · Image-conditioned 3D generation | FID: 19.93 -> 22.55 | Mixed |
| Use SAR3D-NeRF instead of the strongest listed baseline, LGM (V=4) | Image-conditioned 3D generation · Image-conditioned 3D generation | MUSIQ: 54.78 -> 65.77 | Helped |
| Use SAR3D-NeRF instead of the strongest listed baseline, Shap-E | Image-conditioned 3D generation · Image-conditioned 3D generation | COV (%): 61.33 -> 74.17 | Helped |
| Use SAR3D-NeRF instead of the strongest listed baseline, Shap-E | Image-conditioned 3D generation · Image-conditioned 3D generation | MMD (‰): 19.17 -> 13.63 | Helped |
| Use SAR3D-NeRF instead of the fastest listed baseline, Splatter-Image | Image-conditioned 3D generation · Image-conditioned 3D generation | Latency-V100 (s): 0.83 -> 1.64 | Hurt |
| Use SAR3D-Flexicubes instead of SAR3D-NeRF (ablation) | SAR3D · Image-conditioned 3D generation | MUSIQ: 65.77 -> 67.24 | Helped |
Surprising findings
- LGM achieves slightly better FID than SAR3D despite worse MUSIQ and 3D shape quality.
- Not all scales are needed for 3D understanding; dropping the last two scales still yields good captions.
Limitations
- Relies on two separate autoregressive models (generation and understanding) rather than a unified multimodal model.
- Geometry and texture quality limited by volume rendering.
- Scalability not verified due to resource limits.
Experimental setup
Datasets: G-Objaverse 3DTopia captions
Benchmarks: Objaverse-based image-to-3D evaluation (FID, KID, MUSIQ, COV, MMD)
Metrics: FID KID MUSIQ COV MMD latency
Baselines: Splatter-Image OpenLRM One-2-3-45 Lara CRM LGM Shap-E LN3Diff Point-E 3DTopia
Builds on: VAR PointLLM VQVAE LN3Diff LATTE3D
Closest papers by meaning
- AutoPartGen: Autoregressive 3D Part Generation and Discovery (NeurIPS 2025 · similarity 0.89)
- ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding (NeurIPS 2025 · similarity 0.89)
- CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Models (ICML 2026 · similarity 0.88)
- 3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models (NeurIPS 2026 · similarity 0.88)
- PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformers (NeurIPS 2025 · similarity 0.87)
- Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attention (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.