Research map / 3D vision / 3D generation
Large-Vocabulary 3D Diffusion Model with Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiffTF, a triplane-based feed-forward 3D diffusion model with a 3D-aware transformer (shared-weight cross-plane attention) and 3D-aware encoder/decoder, trained as one model for large-vocabulary 3D object generation.
Problem
Generating diverse, high-quality 3D objects across a large vocabulary of categories with a single generative model, where existing 3D generators mostly handle one or a few categories.
Approach
Treat triplane features as a mix of generalized 3D knowledge (extracted via a shared-weight cross-plane attention across xy/yz/xz planes) and specialized per-plane features (self-attention), combined in a 3D-aware transformer inside a diffusion model; a 3D-aware encoder/decoder with cross-plane attention enhances 3D relations before the transformer.
Key results
- On OmniObject3D class-conditional generation, DiffTF reaches FID 25.36 and COV 43.57%, versus 41.56 FID for EG3D and 122.40 FID for NFD w/ texture.
- On ShapeNet single-category unconditional generation, DiffTF improves FID over baselines in Car (36.68 vs 41.41 for GET3D), Plane (14.46 vs 26.80), and Chair (35.16 vs 35.33).
- Ablation shows the 3D-aware transformer lowers FID from 52.35 (original transformer on top of the 3D encoder) to 25.36.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiffTF instead of the strongest reported baseline | OmniObject3D generation · OmniObject3D | FID: 41.56 -> 25.36 | Helped |
| Use DiffTF instead of the strongest reported baseline | ShapeNet generation · ShapeNet Car | COV (%): 39.21 -> 53.25 | Helped |
| Add triplane regularization (ablation) | DiffTF triplane fitting · OmniObject3D | FID: 125.21 -> 113.01 | Helped |
| Add triplane normalization after regularization (ablation) | DiffTF triplane fitting · OmniObject3D | FID: 113.01 -> 109.81 | Helped |
| Add cross-plane attention to the encoder (ablation) | DiffTF · OmniObject3D | FID: 109.81 -> 39.13 | Helped |
| Replace original transformer with 3D-aware transformer (ablation) | DiffTF · OmniObject3D | FID: 52.35 -> 25.36 | Helped |
| Use the original transformer instead of the 3D-aware transformer (ablation) | DiffTF · OmniObject3D | FID: 25.36 -> 52.35 | Hurt |
| Add triplane normalization and regularization (ablation) | DiffTF triplane fitting · OmniObject3D | FID: 125.21 -> 109.81 | Helped |
Surprising findings
- Replacing the 3D-aware transformer with a standard 2D self-attention transformer hurts performance, as the 2D interdependencies break the 3D relations built by the encoder.
Limitations
- Triplane fitting takes about 3 minutes per object, which is time-consuming when scaled to around 1 million objects.
- Details of generated triplanes in some complicated categories still have room for improvement.
Experimental setup
Datasets: ShapeNet OmniObject3D
Benchmarks: ShapeNet OmniObject3D
Metrics: FID KID COV MMD (Chamfer Distance)
Baselines: EG3D GET3D DiffRF NFD w/ texture
Builds on: EG3D DiT NFD DDPM DDIM
Closest papers by meaning
- Diffusion 3D Features (Diff3F): Decorating Untextured Shapes with Distilled Semantic Features (CVPR 2024 · similarity 0.89)
- DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data (CVPR 2024 · similarity 0.89)
- PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformers (NeurIPS 2025 · similarity 0.89)
- One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (CVPR 2024 · similarity 0.88)
- LT3SD: Latent Trees for 3D Scene Diffusion (CVPR 2025 · similarity 0.88)
- Structured 3D Latents for Scalable and Versatile 3D Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.