Research map / Vision-language and multimodal / Human and embodied modeling
3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment
Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, Qing Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pre-train on ScanScribe before fine-tuning (ablation) | 3D-VisTA · Nr3D | overall accuracy (%): 57.5 -> 64.2 | Improved |
| pre-train on ScanScribe before fine-tuning (ablation) | 3D-VisTA · ScanRefer (Mask3D) | overall acc@0.5 (%): 41.5 -> 45.8 | Improved |
| pre-train on ScanScribe before fine-tuning (ablation) | 3D-VisTA · Scan2Cap | CIDEr@0.25: 66.8 -> 71.0; BLEU-4 36.6 -> 36.5 | Mixed |
| pre-train on ScanScribe before fine-tuning (ablation) | 3D-VisTA · SQA3D | average accuracy (%): 46.7 -> 48.5 | Improved |
| add pre-training on 30-40% of annotations vs scratch with full data | 3D-VisTA · ScanRefer / ScanQA | accuracy | Improved |
| increase fusion Transformer depth beyond 4 layers (ablation) | 3D-VisTA · ScanRefer VG | grounding accuracy (%): 57.4 -> 56.3 (8 layers) | No clear effect |
| pre-train with MLM only (ablation) | 3D-VisTA · ScanRefer VG | grounding accuracy (%): 52.0 -> 51.5 | Hurt |
| add masked object modeling (MOM) to MLM (ablation) | 3D-VisTA pre-training · ScanRefer VG | grounding accuracy (%): 51.5 -> 57.1 | Improved |
| add scene-text matching (STM) to MLM + MOM (ablation) | 3D-VisTA pre-training · ScanQA EM@1 | EM@1 (%): 22.5 -> 23.8 | Improved |
| add 3R-Scan and Objaverse object replacement to ScanNet pre-training data (ablation) | 3D-VisTA pre-training · ScanRefer VG | grounding accuracy (%): 54.6 -> 57.4 | Improved |
| use Mask3D instead of PointGroup proposals (ablation) | 3D-VisTA · ScanRefer | overall acc@0.5 (%): 37.3 -> 45.8 | Improved |
Later papers that compared with this method
7 later accepted papers report 3D-VisTA as a baseline in their results tables; 6 of them report a better number on most of the results they share with it.
- PanoGrounder: Bridging 2D and 3D with Panoramic Scene Representations for VLM-based 3D Visual Grounding (ECCV 2026 · better on 1 of 1 shared result)
- Mamba-3VL: Taming State Space Model for 3D Vision Language Learning (ICCV 2025 · better on 4 of 4 shared results)
- 3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer (CVPR 2025 · better on 0 of 1 shared result)
- Situational Awareness Matters in 3D Vision Language Reasoning (CVPR 2024 · better on 5 of 5 shared results)
- An Embodied Generalist Agent in 3D World (ICML 2024 · better on 4 of 4 shared results)
- Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQA (AAAI 2024 · better on 3 of 3 shared results)
- G^3-LQ: Marrying Hyperbolic Alignment with Explicit Semantic-Geometric Modeling for 3D Visual Grounding (CVPR 2024 · better on 3 of 3 shared results)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models (ICLR 2026 · similarity 0.89)
- Unifying 2D and 3D Vision-Language Understanding (ICML 2025 · similarity 0.89)
- G^3-LQ: Marrying Hyperbolic Alignment with Explicit Semantic-Geometric Modeling for 3D Visual Grounding (CVPR 2024 · similarity 0.89)
- Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysis (CVPR 2025 · similarity 0.89)
- MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations (NeurIPS 2024 · similarity 0.89)
- From Thousands to Billions: 3D Visual Language Grounding via Render-Supervised Distillation from 2D VLMs (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
