Research map / Image generation / Text-to-image diffusion
Learning Continuous 3D Words for Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns continuous 3D word tokens (an MLP mapping attribute values to text-embedding tokens) via LoRA fine-tuning of Stable Diffusion on rendered mesh images, enabling slider-style continuous control of illumination, pose, and dolly zoom in text-to-image generation.
Problem
Text-to-image diffusion models cannot precisely control abstract continuous 3D attributes such as illumination direction, non-rigid shape change, object orientation, and camera parameters, since text describes these only coarsely.
Approach
Replace discrete attribute tokens with a continuous function (positional encoding + 2-layer MLP) that maps attribute values to token embeddings, trained in two stages (object identity token first, then attribute MLP) with ControlNet-augmented backgrounds/textures and an [obj] negative prompt at inference.
Key results
- In user study, Continuous 3D Words were preferred in 55.4% of cases on average versus 23.6% for ControlNet (1.0) and 21.0% for ControlNet (0.5).
- Average user ranking of 2.43 ± 0.70 for the proposed method vs 1.84 (ControlNet 1.0) and 1.73 (ControlNet 0.5).
- Continuous words can be trained from a single mesh and generalize to other categories (e.g., horses, taxis, polar bears) and support interpolation, unlike discrete tokens.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Continuous 3D Words instead of the strongest ControlNet baseline | Text-to-image generation · Wing pose | User preference: 28.3% -> 61.7% | Helped |
| Use Continuous 3D Words instead of the strongest ControlNet baseline | Text-to-image generation · Illumination | User preference: 28.8% -> 55.0% | Helped |
| Use Continuous 3D Words instead of the strongest ControlNet baseline | Text-to-image generation · Wing pose + orientation | User preference: 35.0% -> 52.5% | Helped |
| Use Continuous 3D Words instead of the strongest ControlNet baseline | Text-to-image generation · Illumination + orientation | User preference: 32.5% -> 52.5% | Helped |
| Remove two-stage training (ablation) | Continuous 3D Words · Generalizing attributes to objects unlike the training mesh | Prompt and attribute fidelity | Hurt |
| Add two-stage training (ablation) | Continuous 3D Words · Generalizing attributes to objects unlike the training mesh | Prompt and attribute fidelity | Helped |
| Remove ControlNet augmentation (ablation) | Continuous 3D Words · Image generation with varied backgrounds | Background realism | Hurt |
| Add ControlNet augmentation (ablation) | Continuous 3D Words · Image generation with varied backgrounds | Background realism | Helped |
| Remove the object identifier from the negative prompt (ablation) | Continuous 3D Words · Image generation with new objects | Image aesthetics | Hurt |
| Add the object identifier to the negative prompt (ablation) | Continuous 3D Words · Image generation with new objects | Image aesthetics | Helped |
| Replace the continuous MLP with discrete tokens (ablation) | Wing-pose control · Interpolating between attribute values | Attribute fidelity | Hurt |
Surprising findings
- Users sometimes preferred images that more strictly followed the exact conditions even when the bird head or physical plausibility was poor (Figure 9).
Limitations
- Fails when the text prompt specifies a style (e.g., Monet painting).
- Generated objects sometimes overfit to training mesh shapes (e.g., T-Rex standing on four feet).
- Dolly zoom required five chair meshes and was harder to train; ControlNet baseline needed manual selection of conditioning images.
Experimental setup
Datasets: rendered images from single mesh (dog, animated dove) Pix3D chairs
Benchmarks: user study on generated images (wing pose, illumination, combined controls)
Metrics: user preference percentage average user ranking
Baselines: ControlNet (strength 1.0) ControlNet (strength 0.5) Zero-1-to-3 discrete tokens null-text embedding interpolation
Builds on: Textual Inversion Dreambooth ControlNet Stable Diffusion LoRA
Closest papers by meaning
- Controllable Dynamic 3D Shape Generation via 3D Trajectories and Text (NeurIPS 2026 · similarity 0.87)
- Generating compositional scenes via Text-to-image RGBA Instance Generation (NeurIPS 2024 · similarity 0.87)
- Generating Images with 3D Annotations Using Diffusion Models (ICLR 2024 · similarity 0.86)
- 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation (CVPR 2024 · similarity 0.86)
- Generative Photography: Scene-Consistent Camera Control for Realistic Text-to-Image Synthesis (CVPR 2025 · similarity 0.86)
- DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.