Research map / Image generation / Text-to-image diffusion
HyperFields: Towards Zero-Shot Generation of NeRFs from Text
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HyperFields, a dynamic hypernetwork that maps text embeddings to NeRF MLP weights, trained by distilling per-prompt DreamFusion teacher NeRFs, enabling zero-shot and few-step fine-tuned text-to-NeRF generation.
Problem
Text-to-3D synthesis with NeRFs requires computationally expensive per-prompt optimization (SDS or CLIP guidance); the goal is to generate NeRFs from text in a single forward pass, generalizing to unseen prompts.
Approach
A dynamic hypernetwork generates each NeRF MLP layer's weights conditioned on the text token embedding and the previous layer's activations (minibatch-averaged), and is trained via NeRF distillation (photometric loss against frozen per-prompt teacher NeRFs) rather than SDS alone, letting one network fit 100+ scenes.
Key results
- Zero-shot in-distribution NeRFs achieve CLIP top-1 retrieval of 57.1 for unseen prompts vs 69.5 for seen prompts.
- On out-of-distribution prompts, HyperFields reaches CLIP top-3 precision 0.77 and recall 0.63 vs 0.50 and 0.37 for Stable DreamFusion, with KID 0.13 vs 0.17.
- Fine-tuning converges about 5x faster than DreamFusion baselines on average; single network fits over 100 scenes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HyperFields rather than Stable DreamFusion | Out-of-distribution scene generation · Out-of-distribution prompts | CLIP Top-3 precision: 0.50 -> 0.77 | Helped |
| Use HyperFields rather than Stable DreamFusion | Out-of-distribution scene generation · Out-of-distribution prompts | CLIP Top-3 recall: 0.37 -> 0.63 | Helped |
| Use HyperFields rather than Stable DreamFusion | Out-of-distribution scene generation · Out-of-distribution prompts | KID: 0.17 -> 0.13 | Helped |
| Use HyperFields rather than Stable DreamFusion | Out-of-distribution scene generation · Out-of-distribution prompts | SSIM: 0.55 -> 0.62 | Helped |
| Use HyperFields instead of the best DreamFusion baseline | Out-of-distribution scene generation · Six prompts in the user study | average user-reported rank: Golden Blender: 2.5 ± 0.11 -> 1.3 ± 0.14; Yarn Shoes: 2.4 ± 0.13 -> 1.0 ± 0.09; Yarn Skateboard: 2.3 ± 0.11 -> 1.3 ± 0.0 | Helped |
| Fine-tune HyperFields rather than DreamFusion baselines | Out-of-distribution scene generation · Out-of-distribution prompts | convergence speed: 5x speedup | Helped |
| Generate unseen combinations with HyperFields rather than per-scene DreamFusion optimization | In-distribution scene generation · 27 held-out test scenes | generation time: approximately 30 minutes to generate each test scene; less than a minute for all 27 test scenes | Helped |
| Generate unseen in-distribution combinations rather than seen prompts | HyperFields · In-distribution prompts | CLIP retrieval score: Unseen Top-1: 57.1; Seen Top-1: 69.5; Unseen Top-10: 95.2; Seen Top-10: 96.6 | No clear effect |
| Add activation conditioning to the hypernetwork (ablation) | HyperFields · Packing styles including glacier, origami, plaid, and stained glass | style distinction | Helped |
| Add NeRF distillation instead of training the packed dynamic hypernetwork with SDS (ablation) | HyperFields · Packing multiple scenes | geometry diversity | Helped |
| Use the dynamic hypernetwork rather than a static hypernetwork (ablation) | HyperFields · Packing four scenes | style distinction | Helped |
Surprising findings
- Static hypernetwork without activation conditioning collapses distinct styles (glacier/origami, plaid/stained glass) even with only 4 scenes.
- Interpolating raw BERT token embeddings yields non-smooth colour transitions, whereas interpolating at the hypernetwork level is smoother.
Limitations
- Quality is bounded by the teacher models (Stable Diffusion, ProlificDreamer), inheriting their limitations such as struggles with long compositional prompts and Janus-face artifacts.
- Out-of-distribution prompts still require fine-tuning with SDS to reach high quality; zero-shot OOD outputs are nearest-neighbour approximations.
- Evaluation relies on a small set of scenes (9 shapes, 8 colours) and a user study with 450 ratings; no large-scale benchmark.
Experimental setup
Datasets: text prompts for object/attribute combinations teacher NeRFs generated with DreamFusion (Stable Diffusion guidance) ProlificDreamer teacher NeRFs
Benchmarks: CLIP retrieval top-k on held-out scenes out-of-distribution user study CLIP top-3 precision/recall, KID, SSIM on OOD renders
Metrics: CLIP retrieval top-k accuracy average user rank CLIP top-3 precision CLIP top-3 recall KID SSIM convergence speed
Baselines: Stable DreamFusion (scratch) Stable DreamFusion (pretrained-finetuned) ATT3D Shap-E HyperDiffusion
Builds on: DreamFusion Score Distillation Sampling InstantNGP HyperNetworks
Closest papers by meaning
- InsertNeRF: Instilling Generalizability into NeRF with HyperNet Modules (ICLR 2024 · similarity 0.91)
- LLaNA: Large Language and NeRF Assistant (NeurIPS 2024 · similarity 0.88)
- NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild (CVPR 2024 · similarity 0.88)
- NeRF-VPT: Learning Novel View Representations with Neural Radiance Fields via View Prompt Tuning (AAAI 2024 · similarity 0.88)
- TextNeRF: A Novel Scene-Text Image Synthesis Method based on Neural Radiance Fields (CVPR 2024 · similarity 0.88)
- Language-driven Object Fusion into Neural Radiance Fields with Pose-Conditioned Dataset Updates (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.