Research map / Vision-language and multimodal / Multimodal LLMs

Merge3D: Efficient 3D Multimodal LLMs via Joint 2D-3D Token Merging

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Merge3D, a token merging module (SemGeo Merger) for dual-encoder 3D video MLLMs that selects dominant tokens by 2D attention and merges contextual tokens using a multiplicative 2D semantic + 3D geometric similarity, cutting visual tokens by up to 70% with ~3x speedup.

Problem

Dual-encoder 3D video MLLMs produce long multi-view visual token sequences that are costly to process, and existing 2D-only token compression discards spatially critical tokens needed for 3D grounding.

Approach

Merge tokens using a hybrid similarity that multiplies 2D semantic similarity and 3D geometric similarity, with dominant tokens chosen by 2D encoder attention, so merging stays within spatially coherent regions while respecting semantics.

Key results

• At 5% visual token retention, Merge3D reaches 57.9 CIDEr on Scan2Cap vs 52.4 for Visionzip-3D, 49.5 for Visionzip-2D and 51.2 for Randomzip, versus 78.6 for the full baseline.
• On CV-Bench at 30% retention Merge3D keeps 79.6% average accuracy vs 82.1% baseline, with about 2.5x speedup (4.14s vs 10.58s).
• At 5% retention on CV-Bench, Merge3D averages 74.8% vs 68.9% for Visionzip-2D and 63.6% for Visionzip-3D.

Surprising findings

• Pure 3D-geometry merging outperforms 2D-semantic merging on grounding (Scan2Cap) while 2D-guided merging is better on spatial reasoning, a task-dependent pattern.
• Merge3D's gains over baselines are modest at 30% retention and the Multi-View BLINK subset stays around 55-57% for all methods.

Limitations

• Multi-View BLINK subset remains challenging for all compression methods, suggesting token merging alone is insufficient for large viewpoint changes.
• Evaluated only on VG LLM (Qwen2.5-VL-based) backbone; the authors plan to extend to more backbones.
• Speedup is sub-quadratic relative to token reduction, with tokenization and data movement overhead.

Experimental setup

Datasets: ScanRefer Scan2Cap SPAR-7M LLaVA-Video-178K

Benchmarks: Scan2Cap CV-Bench BLINK

Metrics: CIDEr BLEU-4 METEOR ROUGE-L accuracy inference time speedup

Baselines: VisionZip-2D VisionZip-3D Randomzip VG LLM-4B Video-3D LLM LLaVA-3D LEO Chat-Scene

Builds on: VG LLM VisionZip VGGT

Closest papers by meaning

• 3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models (NeurIPS 2026 · similarity 0.90)
• Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction–Reasoning Synergy (ICLR 2026 · similarity 0.90)
• Let Language Constrain Geometry: Vision–Language Models as Semantic and Spatial Critics for 3D Generation (ICML 2026 · similarity 0.89)
• Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering (NeurIPS 2026 · similarity 0.89)
• PPE: Positional Preservation Embedding for Token Compression in Multimodal Large Language Models (ICLR 2026 · similarity 0.88)
• MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentation (NeurIPS 2025 · similarity 0.88)