Research map / Image generation / Text-to-image diffusion
TextField3D: Towards Enhancing Open-Vocabulary 3D Generation with Noisy Text Fields
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TextField3D, a conditional 3D generator that conditions on noisy text fields (NTFGen with dynamic noise injection) plus a view-invariant text-3D binding (NTFBind) to improve open-vocabulary text-to-3D generation trained on limited 3D data.
Problem
Text-to-3D generators trained on limited text-3D data have restricted vocabulary and text control, often collapsing to stereotyped concepts for certain prompts and losing open-vocabulary generation ability.
Approach
Instead of conditioning on a single text embedding, inject dynamic noise into CLIP text features to form a noisy text field covering a range of objects, and bind text to 3D via view-invariant contrastive features, enabling real-time generation from limited data.
Key results
- Under CLIP R-Precision in Table 1, the method substantially outperforms prior 3D-supervised methods and approaches V-L pre-trained methods.
- On ShapeNet55 with BLIP-2 captions, the method reaches FID 26.94 and CLIP-Score 30.35, versus 29.02 / 29.80 for the reproduced variant (b) and 41.66 FID for GET3D.
- On a 415-prompt threestudio list, 9-shot generation reaches 37.83% ViT-B/32 R-Precision vs 13.98% for Shap-E (300M).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 9-shot generation instead of 1-shot generation | TextField3D · 415-prompt test list | CLIP R-Precision (ViT-B/32): 19.04% -> 37.83% | Helped |
| Use 9-shot generation instead of 1-shot generation | TextField3D · 415-prompt test list | CLIP R-Precision (ViT-L/14): 24.33% -> 42.41% | Helped |
| Use TextField3D instead of Shap-E (300M) | TextField3D · 415-prompt test list | CLIP R-Precision (ViT-B/32): 13.98% -> 19.04% | Helped |
| Use TextField3D instead of Shap-E (300M) | TextField3D · 415-prompt test list | CLIP R-Precision (ViT-L/14): 19.51% -> 24.33% | Helped |
| Use view-invariant features instead of vanilla CLIP features | Image retrieval features · Same-category instance retrieval, ground-truth images | Retrieval precision: 45.60% -> 59.04% | Helped |
| Use view-invariant features instead of vanilla CLIP features | Image retrieval features · Same-category instance retrieval, generated images | Retrieval precision: 44.84% -> 57.34% | Helped |
| Replace dynamic noise with static noise σ=0.016 (ablation) | TextField3D · ShapeNet55 with BLIP-2 captions | FID: 26.94 -> 27.64 | Hurt |
| Replace dynamic noise with static noise σ=0.016 (ablation) | TextField3D · ShapeNet55 with BLIP-2 captions | CLIP-Score (ViT-B/32): 30.35 -> 30.06 | Hurt |
| Fine-tune GET3D with TAPS3D instead of using TextField3D | ShapeNet55 text-to-3D generation · ShapeNet55 | FID: 26.94% -> 35.12% | Hurt |
| Fine-tune GET3D with TAPS3D instead of using TextField3D | ShapeNet55 text-to-3D generation · ShapeNet55 | CLIP-Score (ViT-B/32): 30.35% -> 28.38% | Hurt |
Surprising findings
- Original TAPS3D fine-tuned on pre-trained GET3D performs worse than the reproduced baseline variant, suggesting fine-tuning strategies depend heavily on the pre-trained single-category generator.
Limitations
- Generation quality degrades as prompts include more concepts; concept mixing is hard with limited training data.
- Limited imagination: e.g., a chair described as avocado-like is rendered as a chair with avocado color.
- Possible regional content replication from training data, though authors argue it is not significant overfitting.
Experimental setup
Datasets: ShapeNet55 Objaverse
Benchmarks: DreamField COCO 153-prompt test subset ShapeNet55 captions
Metrics: CLIP R-Precision FID CLIP-Score (ViT-B/32) view-invariant retrieval precision
Baselines: Shap-E Point-E GET3D SDFusion TAPS3D DreamFusion
Builds on: GET3D CLIP Dream Fields TAPS3D
Closest papers by meaning
- MARVEL-40M+: Multi-Level Visual Elaboration for High-Fidelity Text-to-3D Content Creation (CVPR 2025 · similarity 0.87)
- PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusion (CVPR 2024 · similarity 0.87)
- DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data (CVPR 2024 · similarity 0.87)
- Progressive3D: Progressively Local Editing for Text-to-3D Content Creation with Complex Semantic Prompts (ICLR 2024 · similarity 0.87)
- Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation (ICLR 2024 · similarity 0.86)
- Let Language Constrain Geometry: Vision–Language Models as Semantic and Spatial Critics for 3D Generation (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.