Research map / Vision-language and multimodal / Multimodal LLMs

Revisiting Multimodal Positional Encoding in Vision–Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Systematically analyzes multimodal RoPE (position design and frequency allocation) for VLMs and proposes MHRoPE and MRoPE-Interleave with a spatial-reset mechanism, evaluated by SFT of a Qwen2.5-7B-based VLM on ~2M samples.

Problem

Multimodal positional encodings for vision-language models lack systematic study; existing RoPE extensions (MRoPE, VideoRoPE, HoPE, CircleRoPE, IL-RoPE, Omni-RoPE) suffer from modality confusion, poor cross-modal interval, restricted frequency bands, and incompatibility with text-only RoPE.

Approach

Derives three guidelines (positional coherence, full frequency allocation, preservation of textual priors) and proposes MHRoPE (dedicating attention heads to t/h/w axes) and MRoPE-I (round-robin interleaved channel allocation), both combined with a spatial-reset position design that resets spatial indices for visual content.

Key results

• MRoPE-I beats vanilla RoPE by +2.67 on MMMU, +5.28 on ChartQA, and +3.27 on RefCOCO val (7B setting).
• On Qwen3-VL-8B, MRoPE-I reaches grounding average 75.46 vs 70.09 for vanilla RoPE and 70.97 for MRoPE.
• Frequency allocation ablation: Interleave achieves overall 64.95 vs 63.07-63.31 for VideoRoPE-like and IL-RoPE-like allocations.

Surprising findings

• Diagonal layout (VideoRoPE) causes repetitive nonsensical generation (e.g., '1111...') even when applied only at inference, attributed to position-ID overlap.
• Scaling the rotary base for spatial axes, a seemingly minor change, consistently degrades image benchmark performance.

Limitations

• MHRoPE underperforms MRoPE-I slightly and complicates tensor-parallel distributed training.
• Training data is proprietary-scale 2M SFT samples without public release, limiting reproduction.
• Results are reported with single-run differences often small (e.g., <1 point on many benchmarks).

Experimental setup

Datasets: approx. 2M SFT samples (image captioning, OCR, visual reasoning, grounding, document, long video, multi-turn dialogue) ChartQA

Benchmarks: MMMU MMBench MMStar OCRBench AI2D RealWorldQA DocVQA TextVQA InfoVQA ChartQA BLINK MVBench

Metrics: accuracy mIoU/grounding accuracy

Baselines: Vanilla RoPE MRoPE VideoRoPE HoPE CircleRoPE V2PE IL-RoPE Omni-RoPE

Builds on: RoPE MRoPE V2PE VideoRoPE YaRN

Closest papers by meaning

• HoPE: Hybrid of Position Embedding for Long Context Vision-Language Models (NeurIPS 2025 · similarity 0.92)
• Towards Long-window Anchoring in Vision-Language Model Distillation (AAAI 2026 · similarity 0.90)
• Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Vision-Language Models (ICML 2026 · similarity 0.89)
• PPE: Positional Preservation Embedding for Token Compression in Multimodal Large Language Models (ICLR 2026 · similarity 0.89)
• Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.88)
• Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.88)