Research map / 3D vision / NeRF and radiance fields
LLaNA: Large Language and NeRF Assistant
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds LLaNA, an MLLM that ingests NeRF MLP weights via a frozen nf2vec meta-encoder and a linear projector into LLaMA 2, trained on an auto-annotated ShapeNeRF-Text dataset, and outperforms image- and point-cloud-based MLLMs on NeRF captioning and Q&A.
Problem
Existing MLLMs handle images and point clouds, but NeRFs must be rendered or converted to discrete 2D/3D data to be used, losing information and adding overhead. The paper asks whether an LLM can directly understand NeRFs.
Approach
Process NeRF MLP weights directly with a pretrained meta-network encoder (nf2vec), project the global embedding into an LLM's token space as a single special-token-wrapped input, and train with two-stage projector then instruction tuning on automatically annotated NeRF-text data.
Key results
- On ShapeNeRF-Text detailed captioning, LLaNA-7b reaches 77.43 Sentence-BERT vs 59.02 for the best frozen baseline PointLLM-7b.
- On single-round Q&A, LLaNA-7b achieves 81.03 Sentence-BERT and 50.15 METEOR, above PointLLM-7b (74.70 and 39.76).
- Zero-shot NeRF classification accuracy of LLaNA-7b is 67.14% for trained baselines comparison, above PointLLM at 49.69%; frozen LLaVA multi-view was reported as best zero-shot.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Process NeRF weights directly instead of an image or point cloud | LLaNA-7b versus frozen foundation-model baselines · ShapeNeRF-Text brief captioning | S-BERT: 61.00 -> 68.63 | Helped |
| Process NeRF weights directly instead of an image or point cloud | LLaNA-7b versus frozen foundation-model baselines · ShapeNeRF-Text brief captioning | BLEU-1: 22.84 -> 20.64 | Hurt |
| Process NeRF weights directly instead of an image or point cloud | LLaNA-7b versus frozen foundation-model baselines · HST brief captioning | S-BERT: 57.11 -> 59.20 | Helped |
| Process NeRF weights directly instead of an image or point cloud | LLaNA-7b versus frozen foundation-model baselines · HST brief captioning | BLEU-1: 15.94 -> 9.47 | Hurt |
| Process NeRF weights directly instead of an image or point cloud | LLaNA-7b versus frozen foundation-model baselines · ShapeNeRF-Text detailed captioning | S-BERT: 60.21 -> 77.43 | Helped |
| Process NeRF weights directly instead of an image or point cloud | LLaNA-7b versus frozen foundation-model baselines · ShapeNeRF-Text single-round Q&A | S-BERT: 74.70 -> 81.03 | Helped |
| Render a back view rather than a front view | Image-based MLLM baselines · ShapeNeRF-Text brief captioning | S-BERT: 61.00 -> 54.35 | Hurt |
Surprising findings
- Frozen LLaVA-vicuna-13b with multi-view input is the best zero-shot NeRF classifier, ahead of LLaNA.
- Training the baselines on ShapeNeRF-Text made LLaVA perform significantly worse on brief captioning, while a language-only LLaMA 2 fine-tuned on the data scores notably lower than LLaNA.
Limitations
- nf2vec encoder is trained only on synthetic ShapeNet data and may not generalize to real-world objects.
- Only MLP-based NeRFs are supported; more complex architectures such as InstantNGP are not handled.
- Evaluated only on object-centric NeRFs, not scene-level NeRFs.
Experimental setup
Datasets: ShapeNeRF-Text ShapeNet GPT2Shape Human Shape Text (HST) Objaverse
Benchmarks: ShapeNeRF-Text HST Objaverse subset
Metrics: Sentence-BERT similarity SimCSE similarity BLEU-1 ROUGE-L METEOR accuracy
Baselines: LLaVA-vicuna-13b LLaVA-vicuna-7b BLIP-2 PointLLM-7b GPT4Point 3D-LLM
Builds on: nf2vec LLaVA PointLLM LLaMA 2 Cap3D
Closest papers by meaning
- HyperFields: Towards Zero-Shot Generation of NeRFs from Text (ICML 2024 · similarity 0.88)
- NeRF Analogies: Example-Based Visual Attribute Transfer for NeRFs (CVPR 2024 · similarity 0.88)
- Language-driven Object Fusion into Neural Radiance Fields with Pose-Conditioned Dataset Updates (CVPR 2024 · similarity 0.87)
- NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints (NeurIPS 2025 · similarity 0.87)
- InsertNeRF: Instilling Generalizability into NeRF with HyperNet Modules (ICLR 2024 · similarity 0.87)
- Visual Perception by Large Language Model’s Weights (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.