Research map / Vision-language and multimodal / Multimodal LLMs

Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Vision-Language Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Circle-RoPE, which maps image-token 2D indices onto an annulus orthogonal to the text position axis (with alternating layers of M-RoPE, AGE) to remove cross-modal RoPE positional bias, measured by a new Per-Token Distance metric.

Problem

RoPE in vision-language models couples text and image position indices in a shared index space, introducing spurious cross-modal relative-position bias that can hurt multimodal alignment and grounding.

Approach

Introduce Per-Token Distance (PTD) to measure variance of RoPE distances from each text token to image tokens; remap image token coordinates onto an annulus whose normal aligns with the text index axis (cone-like geometry), so every text token is equidistant to all image tokens (PTD=0), and alternate these layers with standard M-RoPE layers (AGE).

Key results

• On Qwen2.5-VL-3B SFT, Circle-RoPE (alpha=0.5, radius 10) improves average score from 57.46 to 58.46 across five benchmarks, with AI2D gaining +2.58.
• With PTD=0, Circle-RoPE scores 57.76 average versus 54.67 for Unordered PTD=0 and 56.76 for M-RoPE baseline in the strategy ablation.
• On LLaVA-OneVision-Qwen2-0.5B, Circle-RoPE averages 27.36 versus 26.95 for 1D-RoPE and 26.93 for M-RoPE.

Surprising findings

• Unordered embedding satisfies PTD=0 yet degrades performance below the M-RoPE baseline, indicating that equidistance alone is insufficient without preserved image geometry.
• Circle-RoPE initially lags the SFT baseline at 3k steps (adaptation cost) and only surpasses it later in training.

Limitations

• Evaluated only on static 2D image inputs; video and higher-dimensional extension left to future work.
• Gains are modest (about +1 average point) and reported on a single seed setting without variance estimates shown.
• Training from scratch was beyond compute budget, so only SFT-based adaptation of existing RoPE-trained backbones was tested.

Experimental setup

Datasets: MAmmoTH-VL-Sub

Benchmarks: MMMU MMMU-Pro MathVista AI2D RealWorldQA TAM MathVision

Metrics: accuracy Avg score Obj-IoU Func-IoU F1-IoU

Baselines: Qwen2.5-VL (SFT) M-RoPE Unordered embedding LLaVA [1D-RoPE] SAIL-VL InternVL2.5 MiniCPM V-2.6 Phi-3.5 vision

Builds on: RoPE M-RoPE

Closest papers by meaning

• A Circular Argument: Does RoPE need to be Equivariant for Vision? (NeurIPS 2025 · similarity 0.90)
• Revisiting Multimodal Positional Encoding in Vision–Language Models (ICLR 2026 · similarity 0.89)
• HoPE: Hybrid of Position Embedding for Long Context Vision-Language Models (NeurIPS 2025 · similarity 0.89)
• IVC-Prune: Revealing the Implicit Visual Coordinates in LVLMs for Vision Token Pruning (ICLR 2026 · similarity 0.88)
• ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices (CVPR 2025 · similarity 0.88)
• Reading Images Like Texts: Sequential Image Understanding in Vision-Language Models (ICLR 2026 · similarity 0.87)