Research map / Detection and segmentation / Open-set and few-shot detection
RelationField: Relate Anything in Radiance Fields
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends NeRF with a pair-of-rays relationship feature head distilled from GPT-4o set-of-mark relationship descriptions, enabling open-vocabulary relationship queries, 3D scene graph prediction, and relationship-guided instance segmentation.
Problem
Radiance fields and open-vocabulary 3D feature fields capture object-centric semantics but not inter-object relationships, and 3D scene graph methods need explicit 3D data or closed relation sets.
Approach
Represent inter-object relationships as pairs of rays/query locations inside a NeRF, with an extra relationship feature head in CLIP-like language space; supervise it by rendering features distilled from a multimodal LLM (GPT-4o) prompted with set-of-mark, so relationships are multi-view consistent in 3D.
Key results
- On 3DSSG, RelationField reaches object R@5 0.69 and relationship R@50 0.73, outperforming Open3DSG (0.56 / 0.55) and ConceptGraphs (0.37 / 0.69).
- On relationship-guided instance segmentation over ScanNet++ queries, RelationField achieves 0.53 IoU and 0.96 accuracy versus 0.49 IoU and 0.87 accuracy for LangSplat.
- Lifting GPT-4o relationships into 3D beats 2D-only GPT-4 (2D+depth) on 3DSSG relationship recall (0.73 vs 0.52 at R@50).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RelationField rather than Open3DSG | 3D scene graph prediction · 3DSSG RIO10 | Object R@5: 0.56 -> 0.69 | Helped |
| Use RelationField rather than Open3DSG | 3D scene graph prediction · 3DSSG RIO10 | Object R@10: 0.61 -> 0.80 | Helped |
| Use RelationField rather than ConceptGraphs | 3D scene graph prediction · 3DSSG RIO10 | Predicate R@3: 0.74 -> 0.76 | Helped |
| Use RelationField rather than ConceptGraphs | 3D scene graph prediction · 3DSSG RIO10 | Predicate R@5: 0.79 -> 0.82 | Helped |
| Use RelationField rather than ConceptGraphs | 3D scene graph prediction · 3DSSG RIO10 | Relationship R@50: 0.69 -> 0.73 | Helped |
| Use RelationField rather than ConceptGraphs | 3D scene graph prediction · 3DSSG RIO10 | Relationship R@100: 0.71 -> 0.74 | Helped |
| Use RelationField rather than LangSplat for relationship queries | Relationship-guided 3D instance segmentation · ScanNet++ | IoU: 0.49 -> 0.53 | Helped |
| Use RelationField rather than LangSplat for relationship queries | Relationship-guided 3D instance segmentation · ScanNet++ | Accuracy: 0.87 -> 0.96 | Helped |
| Use a 3D-consistent RelationField instead of 2D GPT-4 inference (ablation) | 3D scene graph prediction · 3DSSG RIO10 | Relationship R@50: 0.52 -> 0.73 | Helped |
| Use Llama 3.2 rather than GPT-4 as the relationship extractor (ablation) | RelationField · 3DSSG RIO10 | Object R@10: 0.82 -> 0.80 | Hurt |
| Use Llama 3.2 rather than GPT-4 as the relationship extractor (ablation) | RelationField · 3DSSG RIO10 | Predicate R@3: 0.76 -> 0.75 | Hurt |
| Use Llama 3.2 rather than GPT-4 as the relationship extractor (ablation) | RelationField · 3DSSG RIO10 | Predicate R@5: 0.82 -> 0.80 | Hurt |
Surprising findings
- Llama 3.2 (90B) as the relationship teacher performs only slightly worse than GPT-4o when lifted into the radiance field.
Limitations
- Relationship knowledge depends heavily on the multimodal LLM prompting and its outputs.
- Requires known calibrated camera intrinsics and high-quality multi-view captures.
- Quality is bounded by the underlying radiance field reconstruction.
Experimental setup
Datasets: ScanNet++ LERF Replica 3DSSG (RIO10 subset)
Benchmarks: 3DSSG (RIO10) ScanNet++ relationship-guided instance segmentation
Metrics: top-k recall (R@5, R@10, R@3, R@50, R@100) IoU accuracy
Baselines: GPT-4 (2D+depth) Llama 3.2 (2D+depth) Open3DSG ConceptGraphs LERF OpenNeRF LangSplat
Builds on: NeRF Nerfacto LERF Open3DSG OpenNeRF SAM
Closest papers by meaning
- RelationVGGT : Visual Geometry Transformers for 3D Spatial Relation Segmentation (NeurIPS 2026 · similarity 0.87)
- Rethinking Open-Vocabulary Segmentation of Radiance Fields in 3D Space (AAAI 2025 · similarity 0.87)
- Can Transformers Capture Spatial Relations between Objects? (ICLR 2024 · similarity 0.86)
- OpenHype: Hyperbolic Embeddings for Hierarchical Open-Vocabulary Radiance Fields (NeurIPS 2025 · similarity 0.86)
- GSNeRF: Generalizable Semantic Neural Radiance Fields with Enhanced 3D Scene Understanding (CVPR 2024 · similarity 0.85)
- Latent Radiance Fields with 3D-aware 2D Representations (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.