Research map / Image generation / Text-to-image diffusion
YouDream: Generating Anatomically Controllable Consistent Text-to-3D Animals
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates anatomically controllable text-to-3D animals by guiding NeRF score distillation with a pose-controlled ControlNet (TetraPose) and automatically adapts a 16-pose library via a multi-agent GPT-4o pipeline.
Problem
Text-to-3D diffusion methods produce anatomically inconsistent animals (Janus head, extra or missing limbs) and text cannot specify unusual creatures or precise skeletons; creating 3D pose priors for animals manually is laborious.
Approach
Use a TetraPose ControlNet conditioned on 2D projections of a 3D animal skeleton inside SDS-based NeRF optimization, with control/guidance scheduling, plus a multi-agent GPT-4o (Finder, Observer, Modifier) that edits poses from a 16-animal library to represent new animals.
Key results
- User study: YouDream preferred for naturalness and text-image alignment with roughly 60-62% of 1408 preferences.
- CLIP score 30.86 for YouDream vs 29.78 for MVDream (ViT-B/32).
- Generated novel poses judged to represent the requested animal 91% of the time in the pose user study.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use YouDream instead of MVDream | Text-to-3D animal generation · 22-prompt user study | CLIP score: 29.78 -> 30.86 | Helped |
| Use YouDream rather than the compared models | Text-to-3D asset generation · 22 prompts, 32 participants | user preference (Naturalness and Text-Image alignment): 60-62% | Helped |
| Use the multi-agent LLM pose generator | Animal 3D pose generation · 16 requested animal poses | participant agreement: 91% | Helped |
| Add an initial shape (ablation) | YouDream NeRF generation with pose control · Elephant generation | geometry cleanliness | Helped |
| Add pose control (ablation) | YouDream NeRF generation with initial shape · Elephant generation | 3D consistency | Helped |
| Add initial shape (ablation) | YouDream generation with pose control · Animal generation | geometry cleanliness | Helped |
| Add pose control (ablation) | YouDream generation · Elephant generation | anatomical consistency | Helped |
| Use both guidance scaling and control scaling (ablation) | YouDream training schedule · Animal generation | visual quality | Helped |
| Add guidance scaling (ablation) | YouDream training schedule with control scaling · Animal generation | color naturalness | Helped |
| Add control scaling (ablation) | YouDream training schedule with guidance scaling · Animal generation | color naturalness | Helped |
| Use cosine guidance scheduling and linear control scheduling instead of the proposed schedules (ablation) | YouDream training schedule · Animal generation | texture quality | Hurt |
| Use cosine scheduling for both guidance and control instead of the proposed schedules (ablation) | YouDream training schedule · Animal generation | color quality | Hurt |
Surprising findings
- Single-LLM pose generation performed much worse than the multi-agent setup, and LLMs do not produce good poses from text alone without an input pose to adapt.
- OpenPose ControlNet produces artificial-looking animal images, while TetraPose ControlNet follows unnatural human-on-all-fours-derived poses cleanly.
Limitations
- Uses a low-resolution 128x128 NeRF, limiting sharpness and texture detail versus baselines.
- Unusually shaped animals may not be satisfactorily generated by the multi-agent LLM and may need manual pose editing.
- Longer animation generation is resource-expensive and time-consuming.
Experimental setup
Datasets: AwA-pose Animal Kingdom
Benchmarks: user study (22 prompts, 32 participants) CLIP score (ViT-B/32)
Metrics: user preference (naturalness) user preference (text-image alignment) CLIP score pose agreement rate
Baselines: HiFA Fantasia3D 3DFuse MVDream OpenPose ControlNet 3DFauna
Builds on: DreamFusion ControlNet MVDream Instant-NGP
Closest papers by meaning
- Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior (CVPR 2024 · similarity 0.85)
- How to Move Your Dragon: Text-to-Motion Synthesis for Large-Vocabulary Objects (ICML 2025 · similarity 0.85)
- Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting (NeurIPS 2024 · similarity 0.84)
- UniMC: Taming Diffusion Transformer for Unified Keypoint-Guided Multi-Class Image Generation (ICML 2025 · similarity 0.84)
- Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation (CVPR 2024 · similarity 0.84)
- BEAST3D: animal behavioral analysis and neural encoding from multi-view video via Gaussian splatting (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.