Research map / Video / Video-language understanding
VideoRoPE: What Makes for Good Video Rotary Position Embedding?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes four properties needed for video RoPE and proposes VideoRoPE (3D structure, low-frequency temporal allocation, diagonal layout, adjustable temporal spacing), fine-tuned from Qwen2-VL-7B/Qwen2-7B and shown to beat M-RoPE and other RoPE variants on long video understanding, retrieval, and hallucination benchmarks.
Problem
Extending 1D rotary position embedding to video's spatio-temporal structure in Video LLMs is unresolved; existing RoPE variants mishandle frequency allocation, spatial symmetry, and temporal index scaling, and are distracted by periodic structure.
Approach
Identify four design criteria for video RoPE and propose a 3D RoPE that allocates low-frequency dimensions to time, interleaves x/y in high-frequency dimensions, lays out positions along a diagonal for spatial symmetry, and scales temporal indices with a tunable factor delta; also introduces V-NIAH-D, a retrieval benchmark with periodic distractors.
Key results
- On V-NIAH-D, VideoRoPE reaches 87.11 average accuracy versus 74.67 for M-RoPE; on V-NIAH it reaches 91.11 vs 78.67.
- At 64k context, VideoRoPE scores 57.26 on LongVideoBench vs 54.35 for M-RoPE, 65.56 on MLVU vs 61.10, and 61.33 on Video-MME vs 59.67.
- On VideoHallucer, VideoRoPE averages 46.2 versus 34.3 for M-RoPE and 39.0 for TAD-RoPE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VideoRoPE instead of M-RoPE | Video LLM · V-NIAH | accuracy: 78.67 -> 91.11; +12.4 | Helped |
| use VideoRoPE instead of M-RoPE | Video LLM · V-NIAH-D | accuracy: 74.67 -> 87.11; +12.4 | Helped |
| use VideoRoPE instead of M-RoPE | Video LLM · LongVideoBench, 64k | score: 54.35 -> 57.26; +2.91 | Helped |
| use VideoRoPE instead of M-RoPE | Video LLM · MLVU, 64k | score: 61.10 -> 65.56; +4.46 | Helped |
| use VideoRoPE instead of M-RoPE | Video LLM · Video-MME, 64k | score: 59.67 -> 61.33; +1.66 | Helped |
| use VideoRoPE instead of M-RoPE | Video LLM · VideoHallucer | Avg.: 34.3 -> 46.2 | Helped |
| add Diagonal Layout (DL) to M-RoPE baseline (ablation) | VideoRoPE · MLVU | score: 61.56 -> 63.03 | Helped |
| add Diagonal Layout (DL) to M-RoPE baseline (ablation) | VideoRoPE · V-NIAH-D | accuracy: 74.67 -> 76.44 | Helped |
| add Low-frequency Temporal Allocation (LTA) to DL (ablation) | VideoRoPE · LongVideoBench, 64k | score: 53.63 -> 55.60 | Helped |
| add Adjustable Temporal Spacing (ATS) to DL and LTA (ablation) | VideoRoPE · MLVU, 64k | score: 63.26 -> 65.56 | Helped |
| interleave x and y allocation rather than using sequential allocation (ablation) | VideoRoPE · LongVideoBench, 64k | score: 54.77 -> 57.26 | Helped |
| use low-frequency temporal allocation [xy...t...] rather than [t t x y x y] (ablation) | VideoRoPE · LongVideoBench Avg | score: 59.06 -> 60.14 | Helped |
Surprising findings
- M-RoPE's attention locates the needle mainly through vertical position rather than temporal position, and it still fails the multiple-choice question under distractors.
- Vanilla RoPE and TAD-RoPE score only about 30% on V-NIAH, far below the 3D RoPE variants.
Limitations
- Experiments fine-tune only from Qwen2-VL-7B / Qwen2-7B, so generality to other backbones is not shown.
- V-NIAH-D uses a single fixed distractor period of 200 frames, a narrow stress test.
- Ablations and comparisons are reported on a single training recipe with limited seeds; gains on some benchmarks are small (e.g., +1.7 Video-MME).
Experimental setup
Datasets: LLaVA-Video-178k
Benchmarks: LongVideoBench MLVU Video-MME V-NIAH V-NIAH-D VideoHallucer
Metrics: accuracy
Baselines: Vanilla RoPE TAD-RoPE M-RoPE
Builds on: RoPE M-RoPE TAD-RoPE RoPE-Tie V-NIAH
Closest papers by meaning
- ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices (CVPR 2025 · similarity 0.86)
- A Circular Argument: Does RoPE need to be Equivariant for Vision? (NeurIPS 2025 · similarity 0.86)
- HoPE: Hybrid of Position Embedding for Long Context Vision-Language Models (NeurIPS 2025 · similarity 0.84)
- Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Vision-Language Models (ICML 2026 · similarity 0.84)
- TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning (ICLR 2025 · similarity 0.83)
- Base of RoPE Bounds Context Length (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.