Research map / Point clouds / Point cloud classification
LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding Reasoning and Planning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LL3DA, a 3D large multimodal assistant that takes point clouds plus text and visual prompts (clicks/boxes) through a Q-Former-style interactor into a frozen OPT-1.3B, and instruction-tunes it for 3D dense captioning, 3D QA, and scene description.
Problem
Enabling large multimodal models to understand, reason, and plan in complex 3D scenes directly from permutation-invariant point clouds, while resolving textual ambiguity through visual interactions, without costly multi-view feature projection.
Approach
Aggregates text instructions, user clicks/box prompts, and frozen 3D scene embeddings into a fixed set of 32 learnable queries via a multi-modal transformer (early fusion), then feeds these as a prefix to a frozen LLM, so the LLM sees interaction-aware 3D features.
Key results
- On ScanRefer, LL3DA reaches 65.19 C@0.5 for dense captioning, +3.38 over Vote2Cap-DETR (61.81 on Nr3D baseline comparison aside).
- On Nr3D, LL3DA reaches 51.18 C@0.5, +7.34 over Vote2Cap-DETR.
- On ScanQA test w/ object, LL3DA reaches 78.16 CIDEr, outperforming prior methods; clicks at test time raise ScanQA validation CIDEr from 76.79 to 82.91.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace direct visual-prompt injection with early fusion in the Multi-Modal Transformer (ablation) | LL3DA · ScanRefer | C@0.5: 59.39 -> 62.84; +3.45% | Helped |
| Add auxiliary localization instructions during training (ablation) | LL3DA · ScanRefer | C@0.5: 60.85 -> 61.81 | Helped |
| Add click prompts as an auxiliary training input (ablation) | LL3DA · ScanRefer | C@0.5: 60.85 -> 62.20 | Helped |
| Add both auxiliary localization instructions and click prompts during training (ablation) | LL3DA · ScanRefer | C@0.5: 60.85 -> 62.84 | Helped |
| Add auxiliary answer-and-localize instructions during training (ablation) | LL3DA · ScanQA validation | CIDEr: 67.85 -> 72.73 | Helped |
| Add visual prompts during training (ablation) | LL3DA · ScanQA validation | CIDEr: 67.85 -> 68.09 | No clear effect |
| Add both auxiliary answer-and-localize instructions and visual prompts during training (ablation) | LL3DA · ScanQA validation | CIDEr: 67.85 -> 74.80 | Helped |
| Add textual task instructions (ablation) | LL3DA · ScanRefer | C@0.5: 60.20 -> 62.84 | Helped |
| Add visual prompts on related objects at inference (ablation) | LL3DA · ScanQA validation | CIDEr: 76.79 -> 82.91 | Helped |
| Add visual prompts on related objects at inference (ablation) | LL3DA · ScanQA validation | BLEU-4: 13.53 -> 11.80 | Hurt |
| Use LL3DA instead of Vote2Cap-DETR | 3D dense captioning · ScanRefer | C@0.5: 61.81 -> 65.19; +3.38% | Helped |
| Use LL3DA instead of Vote2Cap-DETR | 3D dense captioning · Nr3D | C@0.5: 43.84 -> 51.18; +7.34% | Helped |
Surprising findings
- Initializing the Q-Former with pre-trained BERT weights led to repetitive outputs, so only word and position embeddings were initialized from BERT.
Limitations
- Generalist model performs poorly on Nr3D because Nr3D and ScanRefer were not differentiated during training.
- Relies on ground-truth-like box predictions from Vote2Cap-DETR as visual prompts for evaluation.
Experimental setup
Datasets: ScanNet ScanRefer Nr3D ScanQA 3D-LLM ScanNet subset
Benchmarks: ScanRefer Nr3D ScanQA
Metrics: CIDEr BLEU-4 METEOR ROUGE-L m@kIoU
Baselines: Scan2Cap MORE 3DJCG 3D-VLP Vote2Cap-DETR ScanQA 3D-LLM 3D-VisTA UniT3D
Builds on: Q-Former 3D-LLM Vote2Cap-DETR OPT
Closest papers by meaning
- 3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer (CVPR 2025 · similarity 0.91)
- Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction–Reasoning Synergy (ICLR 2026 · similarity 0.90)
- VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.90)
- Language-Image Models with 3D Understanding (ICLR 2025 · similarity 0.89)
- Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding (CVPR 2025 · similarity 0.89)
- Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.