Research map / 3D vision / Face and shape
ShapeWords: Guiding Text-to-Image Synthesis with 3D Shape-Aware Prompts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Maps a 3D shape (Point-BERT embedding) into OpenCLIP text-embedding space via a cross-attention Shape2CLIP residual module, trained with SDS on Stable Diffusion 2.1, so text-to-image generation can be guided by full 3D geometry without depth maps.
Problem
Text-to-image diffusion models lack fine-grained, view-independent 3D shape control; existing shape guidance (depth, edge, normal maps) is single-view and often overrides textual context.
Approach
Embeds a 3D shape into the prompt's CLIP token embeddings (shape-identifier and EOS tokens) through a learned cross-attention residual module (Shape2CLIP) conditioned on Point-BERT features, so Stable Diffusion receives shape and text jointly without any depth/edge conditioning; a scalar lambda controls shape-guidance strength.
Key results
- On compositional prompts, ShapeWords reaches FID 73.8 and CLIP 31.5 versus ControlNet's 97.0 and 26.9.
- ShapeWords@40 achieves shape adherence comparable to the original ControlNet on silhouette metrics without depth conditioning.
- In a 250-participant MTurk study, ShapeWords was preferred for prompt adherence (54.4% vs 14.2%) and shape resemblance (57.1% vs 15.7%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ShapeWords instead of ControlNet | Text-to-image generation · Compositional prompts | FID: 97.0 -> 73.8 | Helped |
| Use ShapeWords instead of CNet-Stop@60 | Text-to-image generation · Compositional prompts | FID: 90.5 -> 73.8 | Helped |
| Use ShapeWords instead of CNet-Stop@80 | Text-to-image generation · Compositional prompts | FID: 92.4 -> 73.8 | Helped |
| Use ShapeWords instead of ControlNet | Text-to-image generation · Compositional prompts | CLIP: 26.9 -> 31.5 | Helped |
| Use ShapeWords instead of ControlNet | Text-to-image generation · Compositional prompts | Aes.: 5.24 -> 5.45 | Helped |
| Use ShapeWords instead of ControlNet | Text-to-image generation · Compositional prompts | KID: 10.40 -> 8.58 | Helped |
| Add ShapeWords shape guidance to corresponding CNet-Stop@K variants | ControlNet-Stop variants · Simple prompts | S-IOU | Helped |
| Add ShapeWords shape guidance to corresponding CNet-Stop@K variants | ControlNet-Stop variants · Simple prompts | S-CD | Helped |
| Increase shape guidance strength λ (ablation) | ShapeWords · Shape guidance examples | Shape resemblance | Helped |
Surprising findings
- Depth-conditioned ControlNet prioritizes shape over prompt, and CNet-Stop variants underperform ShapeWords on prompt adherence even at matched shape adherence.
- Guidance is purely geometric: chair and lamp targets produce tables, couches, and bookshelves.
Limitations
- Struggles with fine-grained geometry, especially thin parts.
- Generalization to much more complex prompts would require larger-scale training data.
- Appearance biases such as oversaturation and color preferences from score distillation.
Experimental setup
Datasets: ShapeNet ULIP depth renders 3DILG ShapeNet train split
Benchmarks: ShapeNet simple prompts ShapeNet compositional prompts
Metrics: S-IOU S-CD CLIP score FID KID aesthetics score user study preference
Baselines: ControlNet CNet-Stop@K Ctrl-X
Builds on: Stable Diffusion 2.1 ControlNet Point-BERT Score Distillation Sampling continuous 3D words
Closest papers by meaning
- ShapeWalk: Compositional Shape Editing Through Language-Guided Chains (CVPR 2024 · similarity 0.86)
- Controllable Dynamic 3D Shape Generation via 3D Trajectories and Text (NeurIPS 2026 · similarity 0.86)
- PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusion (CVPR 2024 · similarity 0.86)
- 3D Space as a Scratchpad for Editable Text-to-Image Generation (CVPR 2026 · similarity 0.86)
- Geometry Image Diffusion: Fast and Data-Efficient Text-to-3D with Image-Based Surface Representation (ICLR 2025 · similarity 0.85)
- TexTailor: Customized Text-aligned Texturing via Effective Resampling (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.