Research map / 3D vision / 3D generation
MiKASA: Multi-Key-Anchor & Scene-Aware Transformer for 3D Visual Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MiKASA, an end-to-end trained transformer for 3D visual grounding that adds a self-attention scene-aware object encoder, multi-key-anchor spatial maps, and late fusion of category and spatial scores, reaching state-of-the-art on Sr3D and Nr3D.
Problem
3D visual grounding must match language descriptions to objects in point clouds, but existing methods struggle with object recognition accuracy, multi-anchor descriptions, view-dependent queries, and explainability.
Approach
Encodes object context with a self-attention scene-aware encoder, represents spatial relations relative to every object as a candidate anchor (multi-key-anchor) with attention-based anchor aggregation and view augmentation, and fuses separate category and spatial logits via normalized weighted late fusion for explainable decisions.
Key results
- Achieves 75.2% overall accuracy on Sr3D and 64.4% on Nr3D, the highest reported among compared methods.
- On Nr3D view-dependent subset, reaches 65.4% vs 54.3%-55.1% for prior methods such as MVT and ViewRefer (as reported in Table 1).
- Scene-aware self-attention object encoder raises object classification accuracy from 63.8% (PointNet++) to 70.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MiKASA instead of ViewRefer | 3D visual grounding · Sr3D Overall | accuracy: 67.7% -> 75.4% | Helped |
| use MiKASA instead of ViewRefer | 3D visual grounding · Nr3D Overall | accuracy: 56.8% -> 64.0% | Helped |
| replace the spatial encoder with an MLP for direct object location encoding (ablation) | MiKASA · Nr3D Overall | accuracy: 64.4% -> 45.0% ± 0.4% | Hurt |
| add the feedforward layer (ablation) | MiKASA spatial module · Nr3D Overall | accuracy: 62.4% ± 0.2% -> 64.4% | Helped |
| add text-spatial fusion (ablation) | MiKASA spatial module · Nr3D Overall | accuracy: 62.6% ± 0.1% -> 64.4% | Helped |
| add a GCN-based scene-aware module (ablation) | PointNet++ object encoder · Object classification | accuracy: 63.8% -> 65.5% | Helped |
| add a self-attention-based scene-aware module (ablation) | PointNet++ object encoder · Object classification | accuracy: 63.8% -> 70.8% | Helped |
| replace attention-based spatial aggregation with mean aggregation (ablation) | MiKASA fusion module · Nr3D Overall | accuracy: 64.4% -> 33.9 | Hurt |
| replace attention-based spatial aggregation with max pooling (ablation) | MiKASA fusion module · Nr3D Overall | accuracy: 64.4% -> 61.4% | Hurt |
Surprising findings
- Replacing the attention-based anchor aggregation with mean pooling drops Nr3D accuracy to 33.9%, far below max pooling at 61.4%.
Limitations
- Directional information of objects is lost after encoding, limiting view-dependent accuracy (acknowledged as future work).
- Fusion weights and auxiliary loss coefficients are tuned as hyperparameters, with details deferred to supplementary material.
Experimental setup
Datasets: Nr3D Sr3D
Benchmarks: Sr3D Nr3D
Metrics: accuracy (overall, easy, hard, view-dependent, view-independ
Baselines: ReferIt3D InstanceRefer 3DVG-Transformer SAT MVT BUTD-DETR NS3D ViewRefer
Builds on: MVT BUTD-DETR PointNet++ BERT
Closest papers by meaning
- G^3-LQ: Marrying Hyperbolic Alignment with Explicit Semantic-Geometric Modeling for 3D Visual Grounding (CVPR 2024 · similarity 0.90)
- From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes (NeurIPS 2025 · similarity 0.90)
- Grounded 3D-Aware Spatial Vision-Language Modeling (CVPR 2026 · similarity 0.89)
- Towards CLIP-driven Language-free 3D Visual Grounding via 2D-3D Relational Enhancement and Consistency (CVPR 2024 · similarity 0.89)
- Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding (NeurIPS 2024 · similarity 0.89)
- Multi-Object 3D Grounding with Dynamic Modules and Language-Informed Spatial Attention (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.