Research map / 3D vision / 3D generation
Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KYN, a single-view scene reconstruction method that predicts per-point density using vision-language modulation and language-guided spatial attention, trained self-supervised on KITTI-360.
Problem
Single-view 3D scene reconstruction requires inferring occluded geometry, which existing radiance-field methods predict per point in isolation, leading to inaccurate shapes and trailing artifacts in unobserved regions.
Approach
Enrich each 3D point's features with fine-grained vision-language semantics via multiplicative modulation with category text features, then aggregate features across the scene with linear cross-attention guided by text-based keys so each point's density reflects its semantic and spatial neighborhood.
Key results
- On KITTI-360 scene reconstruction in [4, 50m], KYN reaches Oacc 0.86, IEacc 0.63, IErec 0.73 versus BTS at 0.84, 0.61, 0.53.
- On KITTI-360 object reconstruction in [4, 50m], KYN achieves Oacc 0.75, IEacc 0.64, IErec 0.68 versus BTS at 0.72, 0.61, 0.48.
- Zero-shot on DDAD (trained on KITTI-360), KYN reaches Oacc 0.59 and IErec 0.42 versus BTS at 0.48 and 0.16.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add KYN semantic and spatial reasoning | BTS · KITTI-360 scene 4-20m | IEacc: 0.66 -> 0.70 | Helped |
| add KYN semantic and spatial reasoning | BTS · KITTI-360 scene 4-50m | IErec: 0.53 -> 0.73 | Helped |
| add KYN semantic and spatial reasoning | BTS · KITTI-360 object 4-50m | IErec: 0.48 -> 0.68 | Helped |
| train on KITTI-360 and evaluate zero-shot on DDAD | KYN versus prior methods · DDAD scene 4-50m | Oacc: 0.48 -> 0.59 | Helped |
| add fused VL image features without modulation (ablation) | BTS baseline · KITTI-360 scene 4-50m | Oacc: 0.84 -> 0.84 | No clear effect |
| add VL modulation of image and text features (ablation) | BTS baseline · KITTI-360 scene 4-50m | IErec: 0.53 -> 0.64 | Helped |
| add spatial attention over appearance features (ablation) | BTS baseline · KITTI-360 scene 4-50m | IErec: 0.53 -> 0.60 | Helped |
| add VL modulation to spatial attention (ablation) | spatial-attention variant · KITTI-360 scene 4-50m | IErec: 0.60 -> 0.75 | Helped |
| use VL features in spatial attention instead of ordinary spatial attention (ablation) | KYN spatial-attention variant · KITTI-360 scene 4-50m | IErec: 0.75 -> 0.73 | Mixed |
| replace KYN with 2D semantic feature fusion (LDLS) (ablation) | BTS baseline · KITTI-360 scene 4-50m | IErec: 0.52 -> 0.55 | No clear effect |
| replace KYN with 2D semantic feature fusion (SAFENet) (ablation) | BTS baseline · KITTI-360 scene 4-50m | IErec: 0.52 -> 0.51 | No clear effect |
| broaden future-view supervision from 1s to 1-4s (ablation) | KYN · KITTI-360 scene 4-50m | IErec: 0.73 -> 0.77 | Helped |
Surprising findings
- Naively fusing VL image features without text-based modulation does not improve overall reconstruction accuracy.
- 2D semantic feature fusion methods (LDLS, SAFENet) give no notable gain over the baseline.
Limitations
- Relies on self-supervised multi-view photometric supervision and accumulated LiDAR ground truth.
- Evaluation limited mainly to KITTI-360 with zero-shot test on DDAD only.
Experimental setup
Datasets: KITTI-360 DDAD
Benchmarks: KITTI-360 DDAD
Metrics: Oacc IEacc IErec
Baselines: Monodepth2 PixelNeRF BTS LDLS SAFENet
Builds on: BTS PixelNeRF Monodepth2 LSeg
Closest papers by meaning
- PE3R: Perception-Efficient 3D Reconstruction (CVPR 2026 · similarity 0.91)
- Large Spatial Model: End-to-end Unposed Images to Semantic 3D (NeurIPS 2024 · similarity 0.89)
- MESC-3D:Mining Effective Semantic Cues for 3D Reconstruction from a Single Image (CVPR 2025 · similarity 0.89)
- Tackling View-Dependent Semantics in 3D Language Gaussian Splatting (ICML 2025 · similarity 0.89)
- RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding (CVPR 2024 · similarity 0.88)
- G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.