Research map / Image generation / Text-to-image diffusion
Geometry Image Diffusion: Fast and Data-Efficient Text-to-3D with Image-Based Surface Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GIMDiffusion, which generates multi-chart geometry images plus albedo UV textures with a frozen Stable Diffusion 2.1-based model coupled to a trainable geometry branch via Collaborative Control, yielding relightable textured meshes in under 10 seconds.
Problem
Text-to-3D generation is costly, data-scarce, and hard to make compatible with mesh-based graphics pipelines; proxy representations lose part segmentation and internal structure and require lossy conversion.
Approach
Represent 3D shapes as multi-chart geometry images derived from existing handcrafted UV maps, so generation becomes 2D image diffusion; a frozen pre-trained Text-to-Image model generates albedo UV textures while a trainable clone generates geometry images, linked by Collaborative Control cross-network layers.
Key results
- Generates well-defined 3D meshes in under 10 seconds per object, at speeds comparable to text-to-image models.
- Trained on about 100,000 curated Objaverse objects with a 768x768 working resolution, encoding meshes of up to 589,824 vertices.
- Produces UV layouts that vary significantly across seeds and prompts, unlike the concurrent Yan et al. method, which shows little layout variation within a class.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Reuse pre-trained text-to-image priors through Collaborative Control | GIMDiffusion · Text-to-3D generation beyond the Objaverse training data | generalization | Helped |
| Apply a pre-trained IPAdapter to the frozen base model | GIMDiffusion · Stylized text-to-3D generation | style guidance | Helped |
| Use IPAdapter when the text prompt deviates too much structurally from the conditioning image | GIMDiffusion · Stylized text-to-3D generation | style guidance | Hurt |
Surprising findings
- The frozen base model is not rotationally equivariant, so chart rotations in the UV map degrade quality, notably for human faces.
Limitations
- Visible cracks at chart seams, since charts are not stitched and VAE latent compression hurts small areas.
- Cannot handle arbitrary topology or polygon budget constraints; model sometimes duplicates parts causing z-fighting.
- Generalization depends on frozen base model orientation sensitivity, lowering quality for prompts like faces.
Experimental setup
Datasets: Objaverse Cap3D captions Hong et al. captions
Baselines: Yan et al. geometry image diffusion (concurrent work)
Builds on: Collaborative Control Stable Diffusion Geometry Images XDGAN Multi-Chart Geometry Images
Closest papers by meaning
- Generating Images with 3D Annotations Using Diffusion Models (ICLR 2024 · similarity 0.89)
- PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusion (CVPR 2024 · similarity 0.88)
- DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data (CVPR 2024 · similarity 0.88)
- SweetDreamer: Aligning Geometric Priors in 2D diffusion for Consistent Text-to-3D (ICLR 2024 · similarity 0.88)
- Diffusion 3D Features (Diff3F): Decorating Untextured Shapes with Distilled Semantic Features (CVPR 2024 · similarity 0.88)
- GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.