Research map / 3D vision / 3D generation
Structured 3D Latents for Scalable and Versatile 3D Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TRELLIS, a 3D generator built on a unified Structured LATent (SLAT) representation that pairs a sparse voxel grid with DINOv2 multiview features and decodes to Gaussians, radiance fields, or meshes, using two-stage rectified flow transformers up to 2B parameters.
Problem
Existing 3D generative models lack a unified latent space: mesh/SDF methods capture geometry poorly in appearance, while radiance field and Gaussian latents struggle with accurate surface geometry, and no single latent supports multiple output formats.
Approach
Attach local latents to active voxels on a sparse 64^3 grid, encoded from dense multiview DINOv2 features aggregated per voxel, so a single latent can be decoded into Gaussians, radiance fields, or meshes; generate sparse structure and latents with two rectified flow transformers.
Key results
- On Toys4k text-to-3D, Ours XL reaches CLIP 26.70 and FD_dinov2 237.48, versus 25.56 CLIP and 478.92 FD_dinov2 for the best baseline InstantMesh.
- SLAT reconstruction reaches 32.74 PSNR and 0.0083 CD, outperforming 3DTopia-XL, CLAY, and LN3Diff on all metrics.
- In a user study over 100 participants, the method was strongly preferred over baselines, with about 67% to 94.5% preference depending on baseline and prompt type.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed method instead of InstantMesh | 3D generation · Toys4k text-to-3D | CLIP: 25.56 -> 26.70 | Helped |
| Use the proposed method instead of GaussianCube | 3D generation · Toys4k text-to-3D | FDincep: 27.35 -> 20.48 | Helped |
| Use the proposed method instead of InstantMesh | 3D generation · Toys4k image-to-3D | CLIP: 84.43 -> 85.77 | Helped |
| Use the proposed method instead of InstantMesh | 3D generation · Toys4k image-to-3D | FDincep: 20.22 -> 9.35 | Helped |
| Use SLAT resolution 64 with 8 channels rather than resolution 32 with 16 channels (ablation) | SLAT reconstruction · 3D asset reconstruction | PSNR: 31.64 -> 32.74 | Helped |
| Increase channels from 16 to 32 at resolution 32 (ablation) | SLAT reconstruction · 3D asset reconstruction | PSNR: 31.64 -> 31.80 | Helped |
| Increase channels from 32 to 64 at resolution 32 (ablation) | SLAT reconstruction · 3D asset reconstruction | PSNR: 31.80 -> 31.85 | Helped |
| Replace diffusion with rectified flow in Stage 1 (ablation) | Text-to-3D generation · Training set | FDdinov2: 132.71 -> 113.42 | Helped |
| Replace diffusion with rectified flow in Stage 2 (ablation) | Text-to-3D generation · Training set | FDdinov2: 100.88 -> 95.97 | Helped |
| Increase model size from B to XL (ablation) | Text-to-3D generation · Toys4k | CLIP: 26.47 -> 26.70 | Helped |
| Increase model size from B to XL (ablation) | Text-to-3D generation · Toys4k | FDdinov2: 265.26 -> 237.48 | Helped |
Surprising findings
- Latents trained only with Gaussian decoding faithfully reconstruct meshes and radiance fields when decoders are retrained with the encoder frozen.
Limitations
- Training relies on a large proprietary-curated set of 500K assets and GPT-4o captions, which limits reproducibility.
- Geometry and appearance quality rely on DINOv2 features at 64^3 resolution; coarse voxel structure bounds fine detail.
Experimental setup
Datasets: Objaverse (XL) ABO 3D-FUTURE HSSD Toys4k
Benchmarks: Toys4k Reconstruction on 3D assets (Table 1)
Metrics: PSNR LPIPS Chamfer Distance F-score CLIP score Frechet distance kernel distance user preference
Baselines: Shap-E LGM InstantMesh 3DTopia-XL LN3Diff GaussianCube CLAY
Builds on: rectified flow DINOv2 FlexiCubes Strivec Repaint
Closest papers by meaning
- PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformers (NeurIPS 2025 · similarity 0.89)
- 3DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusion (CVPR 2025 · similarity 0.88)
- Rigel3D: Rig-aware Latents for Animation-Ready 3D Asset Generation (NeurIPS 2026 · similarity 0.88)
- G3DR: Generative 3D Reconstruction in ImageNet (CVPR 2024 · similarity 0.88)
- LT3SD: Latent Trees for 3D Scene Diffusion (CVPR 2025 · similarity 0.88)
- Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materials (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.