Research map / Vision-language and multimodal / Human and embodied modeling
LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LSceneLLM, a 3D-VLM that uses LLM attention maps to select task-relevant regions for dense point-cloud token magnification, and introduces the XR-Scene cross-room benchmark.
Problem
3D vision-language models struggle to locate task-relevant visual information in large, dense multi-room 3D scenes, because object-centric features are task-agnostic and sparse downsampling loses small-object details.
Approach
A dense token selector reuses the LLM's attention from the last text token to sparse scene tokens to pick focus regions, then samples dense point-cloud tokens there and fuses them into the last LLM layers via an adaptive self-attention module (plug-and-play scene magnifier).
Key results
- On XR-QA, LSceneLLM reaches 117.21 CIDEr vs 112.80 for LL3DA under finetuning.
- On ScanQA validation, LSceneLLM achieves 88.24 CIDEr vs 80.20 for Leo (with 145k training data vs 1034k+145k).
- On NuScenes-QA, LSceneLLM reaches 58.8 overall accuracy-like score, about 3.0 points above the previous best generative method.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LSceneLLM instead of Chat-Scene | 3D-VLMs · XR-QA | CIDEr: 114.10 -> 117.21 | Helped |
| Use LSceneLLM instead of Chat-Scene | 3D-VLMs · XR-SceneCaption | CIDEr: 3.58 -> 4.59 | Helped |
| Use LSceneLLM instead of Chat-Scene | 3D-VLMs · XR-EmbodiedPlanning | CIDEr: 46.18 -> 63.08 | Helped |
| Use LSceneLLM instead of Leo | 3D-VLMs · ScanQA validation | CIDEr: 80.20 -> 88.24 | Helped |
| Use LSceneLLM instead of NuscenesQA | outdoor 3D-VLMs · NuscenesQA | Acc: 58.1 -> 56.4 | Hurt |
| Add the scene magnifier module (ablation) | Leo · XR-QA | CIDEr: 110.33 -> 113.46(+3.13) | Helped |
| Add the scene magnifier module (ablation) | Leo · XR-QA-S | CIDEr: 103.16 -> 107.56(+5.29) | Helped |
| Add the scene magnifier module (ablation) | Ll3da · XR-QA | CIDEr: 111.35 -> 115.79(+4.44) | Helped |
| Add the scene magnifier module (ablation) | Ll3da · XR-QA-S | CIDEr: 95.65 -> 106.73(+11.09) | Helped |
| Add the scene magnifier module (ablation) | LSceneLLM · XR-QA | CIDEr: 109.92 -> 117.21(+7.29) | Helped |
| Add the scene magnifier module (ablation) | LSceneLLM · XR-QA-S | CIDEr: 97.57 -> 109.42(+11.85) | Helped |
| Use threshold 127 (AT: 3%-5%) instead of threshold 96 (AT: 10%-20%) (ablation) | LSceneLLM · XR-QA | CIDEr: 117.21 -> 115.92 | Hurt |
Surprising findings
- Zero-shot object-centric models (Chat-Scene, Leo) score far higher on XR-QA than LL3DA but near zero CIDEr on XR-SceneCaption, showing large scene captioning needs region-level modeling.
Limitations
- Dense token selection depends on a threshold and a fixed fraction (10%-20%) of sparse tokens, tuned on XR-QA.
- XR-Scene is generated with GPT-4o annotations, which may introduce noise.
- Gains on outdoor NuScenes-QA are modest and rely on separately trained models.
Experimental setup
Datasets: ScanRefer Nr3D ScanQA 3D-LLM ScanNet subset XR-Scene HM3D SceneVerse NuScenes-QA
Benchmarks: XR-Scene (XR-QA, XR-SceneCaption, XR-EmbodiedPlanning) ScanQA NuScenes-QA
Metrics: CIDEr METEOR ROUGE accuracy
Baselines: LL3DA Leo Chat-Scene Chat3D-v2 3D-LLM SceneLLM OccLLaMA LiDAR-LLM LLaVA
Builds on: LL3DA Chat-Scene Leo 3D-LLM
Closest papers by meaning
- Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding (CVPR 2025 · similarity 0.90)
- SceneScaffold: Active Scene-State Construction for Unified 3D Scene Understanding (NeurIPS 2026 · similarity 0.89)
- Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning (CVPR 2025 · similarity 0.89)
- MLLMs Need 3D-Aware Representation Supervision for Scene Understanding (NeurIPS 2025 · similarity 0.89)
- VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.89)
- Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.