Research map / 3D vision / NeRF and radiance fields
GSN: Generalisable Segmentation in Neural Radiance Field
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distils semantic feature fields (DINO, DINOv2, CLIP, SAM) into the generalisable GNT radiance field so that unseen scenes get per-pixel semantic features and stroke-based multi-view segmentation without per-scene training.
Problem
Scene-specific radiance field semantic distillation (N3F, DFF, ISRF) must be retrained for every new scene, while generalisable radiance fields lack semantic features for segmentation.
Approach
Remove view-direction input from GNT's ray transformer so features are view-agnostic, then use a two-stage scheme: GNT is trained for RGB, and a branched student view/ray transformer head is distilled from 2D teacher feature images (e.g. DINO) across many scenes, enabling feature rendering on unseen scenes.
Key results
- On four LLFF scenes, GSN beats N3F/DFF on all metrics (e.g. Shoerack mean IoU 0.911 vs 0.589 for N3F) and is on par with ISRF (ISRF wins Chesstable 0.912 vs 0.828 mean IoU).
- GSN achieves Stove mean IoU 0.839 vs ISRF 0.827 and Shoerack mean IoU 0.911 vs ISRF 0.861, while being generalisable without per-scene training.
- Segmentation is done with view-agnostic features; removing view direction from the ray transformer improves segmentation quality over original GNT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove view direction as input to the ray transformer (ablation) | GSN · Multi-view segmentation | segmentation quality | Hurt |
| Replace ray-transformer aggregation with volumetric rendering (ablation) | GSN · Multi-view segmentation | segmentation quality | Hurt |
| Slightly vary the number of segmentation clusters from 11 (ablation) | GSN · Segmentation | segmentation quality | No clear effect |
| Use GSN instead of ISRF | GSN · Chesstable | Mean IoU: 0.912 -> 0.828 | Hurt |
| Use GSN instead of ISRF | GSN · Chesstable | Accuracy: 0.990 -> 0.981 | Hurt |
| Use GSN instead of ISRF | GSN · Chesstable | mAP: 0.916 -> 0.833 | Hurt |
| Use GSN instead of ISRF | GSN · Colorfountain | Mean IoU: 0.927 -> 0.927 | No clear effect |
| Use GSN instead of ISRF | GSN · Colorfountain | Accuracy: 0.989 -> 0.989 | No clear effect |
| Use GSN instead of ISRF | GSN · Colorfountain | mAP: 0.927 -> 0.93 | Helped |
| Use GSN instead of ISRF | GSN · Stove | Mean IoU: 0.827 -> 0.839 | Helped |
| Use GSN instead of ISRF | GSN · Stove | Accuracy: 0.992 -> 0.993 | Helped |
| Use GSN instead of ISRF | GSN · Stove | mAP: 0.824 -> 0.835 | Helped |
Surprising findings
- The distilled student features are reported to be cleaner than the teacher DINO features computed on the original images, i.e. the student surpasses its teacher.
- Ray-transformer aggregation outperforms volumetric rendering for distilled semantic features in the generalised setting.
Limitations
- Rendering is slow (about 60 seconds per image) due to the transformer architecture.
- Performance degrades when few source views cover a region, since epipolar geometry prediction becomes unreliable.
- Performs multi-view rather than true 3D segmentation.
Experimental setup
Datasets: IBRNet dataset LLFF NeRF-LLFF Spin-NeRF
Benchmarks: LLFF (Chesstable, Colorfountain, Stove, Shoerack)
Metrics: mean IoU accuracy mAP
Baselines: N3F DFF ISRF GNT with DINO features on rendered images
Builds on: GNT IBRNet ISRF N3F DFF
Closest papers by meaning
- GSNeRF: Generalizable Semantic Neural Radiance Fields with Enhanced 3D Scene Understanding (CVPR 2024 · similarity 0.91)
- Learning Robust Generalizable Radiance Field with Visibility and Feature Augmented Point Representation (ICLR 2024 · similarity 0.88)
- GP-NeRF: Generalized Perception NeRF for Context-Aware 3D Scene Understanding (CVPR 2024 · similarity 0.87)
- Mask-Based Modeling for Neural Radiance Fields (ICLR 2024 · similarity 0.87)
- Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields (CVPR 2024 · similarity 0.86)
- InsertNeRF: Instilling Generalizability into NeRF with HyperNet Modules (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.