Research map / Vision-language and multimodal / Multimodal LLMs

Modeling Cross-vision Synergy for Unified Large Vision Model

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PolyV, a unified large vision model that upcycles Qwen2.5-VL-7B into a sparse MoE with a modality router and trains it with modality-specific pretraining plus coarse (distillation from V-JEPA 2 and VGGT) and fine (scene-graph QA) synergy tuning for image, video, and 3D understanding.

Problem

Existing unified large vision models integrate image, video, and 3D modalities mainly through shared encoders or concatenation, lacking explicit cross-modal synergy in architecture and training, so features remain isolated and transfer of complementary priors is limited.

Approach

A sparse MoE LLM with a top-2 router over four experts, where experts are specialized per modality in stage 1, then a synergy-aware stage 2 distills temporal and 3D priors into learnable synergy tokens and fine-tunes on object/relation-level cross-vision QA built from scene graphs.

Key results

• PolyV reports 71.4 on MMStar, 63.4 on 3DSRBench, and 52.7 on VSI-Bench versus 62.5, 48.4, and 33.0 for its Qwen2.5-VL-7B backbone.
• On ScanQA, PolyV reaches CIDEr 105.6 and on SQA3D EM-1 64.8, outperforming prior 3D models such as ROSS3D (107.0 CIDEr is higher, so PolyV is not best on CIDEr versus ROSS3D's 107.0).
• Ablation shows removing MoE experts drops VSI-Bench from 52.7 to 45.8 and Video-MME from 69.6 to 66.4.

Surprising findings

• Full MoE conversion of all layers gives no clear gain over interval MoE placement while increasing parameters (15.6B total vs 11.3B).
• Coarse-grained and fine-grained tuning each help different benchmark types, with coarse tuning better on 3D benchmarks and fine tuning better on video benchmarks.

Limitations

• Training data is only partially joint across modalities; paired video-image subsets are used rather than full tri-modal data.
• Synergy QA dataset depends on GPT-4o generation and scene graph annotations from limited sources.

Experimental setup

Datasets: LLaVA 1.5-558K LLaVA 1.5-Mix-665K LLaVA-Video-178K ShareGPT4Video LLaVA-3D-Instruct-860K 3RScan Scan2Cap PVSG Action Genome 3DSG

Benchmarks: MMStar 3DSRBench MMSI-Bench VSI-Bench CVBench Video-MME ScanQA SQA3D Open-EQA

Metrics: accuracy BLEU-1 BLEU-4 METEOR ROUGE-L CIDEr EM EM-R LLM-Match

Baselines: GPT-4o Gemini-2.0-Flash LLaVA-OV-7B InternVL3-8B Qwen2.5-VL-7B Qwen3-VL-8B Spatial-MLLM SpaceR LLaVA-3D Video-3D-LLM ROSS3D 3DRS

Builds on: Qwen2.5-VL Uni-MoE V-JEPA 2 VGGT LLaVA-3D

Closest papers by meaning

• MVVBench: Benchmarking 4D Reasoning in Vision-Language Models (NeurIPS 2026 · similarity 0.88)
• Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability (NeurIPS 2026 · similarity 0.87)
• VeXKD: The Versatile Integration of Cross-Modal Fusion and Knowledge Distillation for 3D Perception (NeurIPS 2024 · similarity 0.86)
• 4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities (NeurIPS 2024 · similarity 0.85)
• CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.85)
• LVLM-Aided Alignment of Task-Specific Vision Models (CVPR 2026 · similarity 0.85)