Research map / Vision-language and multimodal / Multimodal LLMs

Proxy3D: Efficient 3D Representations for Vision-Language Models via Semantic Clustering and Alignment

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proxy3D compresses video-frame features into a few hundred semantically clustered 3D proxy tokens (from VGGT point maps and SAM 2 masks) and fine-tunes Qwen2.5-VL-7B with a multi-stage curriculum on the 318K SpaceSpan dataset for efficient 3D VQA, grounding, and spatial reasoning.

Problem

Vision-language models for 3D spatial reasoning either use pixel-aligned or correspondence-based visual tokens that are long and spatially inconsistent, or representation-based 3D tokens that are costly to serialize; the open question is how to build a compact yet accurate 3D visual token sequence.

Approach

Cluster per-patch features within each semantic (SAM 2) group, weighted by 3D point maps from VGGT, into a small number of proxies allocated proportionally to group size; serialize them by BFS, add 3D RoPE and Fourier positional embeddings, and align with an LLM via identifier/semantic embedding stages and a progressive four-stage curriculum.

Key results

• Proxy3D with Qwen2.5-VL-7B uses 700 visual tokens and reaches 93.6 CIDEr on ScanQA and 84.0 Uni@0.5 on ScanRefer, competitive with 8000-token correspondence methods.
• On VSI-Bench, Proxy3D averages 47.0, second only to Spatial-MLLM-4B (48.4), with 450 tokens.
• Semantic grouping raises ScanRefer Uni@0.5 from 57.0 to 82.7 in the ablation, while sequence length 450 vs 3,096 for Spatial-MLLM.

Surprising findings

• Proxy3D performs well without inter-frame cross attention in the vision encoder, unlike correspondence-based methods that rely on it.
• Dense captioning (Scan2Cap) is weaker for representation-based models than correspondence-based ones.

Limitations

• Gap to human-level spatial reasoning remains large, especially on appearance order and route planning.
• Scan2Cap dense captioning underperforms correspondence-based models.
• Performance lags Spatial-MLLM on several VSI-Bench tasks, which may benefit from reward learning (GRPO) not used here.

Experimental setup

Datasets: ScanQA SQA3D ScanRefer Multi3DRefer Scan2Cap MMScan SR-91K VSI-Bench

Benchmarks: ScanQA SQA3D ScanRefer Multi3DRefer Scan2Cap VSI-Bench

Metrics: CIDEr BLEU-4 METEOR ROUGE EM Accuracy@0.25 Accuracy@0.5 F1@0.25 F1@0.5 mean relative accuracy

Baselines: Qwen2-VL-7B GPT4Scene Video-3D-LLM Spatial-MLLM-4B 3DRS LLaVA-3D LEO-VL Chat-Scene Descrip3D GPT-4o Gemini-1.5 Pro

Builds on: Qwen2.5-VL VGGT SAM 2 Chat-Scene MMScan

Closest papers by meaning

• Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning (CVPR 2026 · similarity 0.91)
• VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.90)
• Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction–Reasoning Synergy (ICLR 2026 · similarity 0.89)
• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.89)
• UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding (ICLR 2026 · similarity 0.89)
• Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding (CVPR 2025 · similarity 0.89)