Research map / Vision-language and multimodal / Multimodal LLMs
HoPE: Hybrid of Position Embedding for Long Context Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HoPE, a multimodal RoPE that zeroes the lowest temporal frequencies (hybrid frequency allocation) and scales visual temporal indices dynamically in training and inference, improving long-video understanding and retrieval of Qwen2-based VLMs.
Problem
VLMs degrade on long-context inputs, especially long videos, and existing multimodal RoPE frequency allocations for 3D (t,x,y) positions rely on heuristics and may fail to preserve semantic similarity over long contexts.
Approach
Allocate high frequencies to interleaved x,y and set the lowest frequencies for temporal dimension to zero (NoPE-style), justified by a semantic-preference theory; add dynamic bidirectional temporal scaling factors sampled during training and selected at inference.
Key results
- On V-NIAH, HoPE reaches 63.56 average accuracy vs 52.00 for the best baseline (VideoRoPE), a reported 22.23% relative gain.
- With Qwen2-7B-Video at 32k, HoPE beats vanilla RoPE by 2.82 (MLVU), 4.05 (LongVideoBench) and 1.45 (Video-MME) points.
- Ablation on Video-MME shows hybrid frequency allocation adds about 1.69 average points across context lengths, with DTS giving further gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HoPE instead of the strongest reported baseline | Qwen2-7B-Video · MLVU 32k | score: 62.51 -> 63.85 | Helped |
| Use HoPE instead of the strongest reported baseline | Qwen2-7B-Video · LongVideoBench 32k | score: 53.82 -> 55.34 | Helped |
| Use HoPE instead of the strongest reported baseline | Qwen2-7B-Video · Video-MME 32k | score: 59.13 -> 59.44 | Helped |
| Use HoPE instead of the strongest reported baseline | VLM · V-NIAH | accuracy: 52.00 -> 63.56 | Helped |
| Increase temporal allocation from 16 to 32 dimensions, reducing spatial allocations from 24 each to 16 each (ablation) | HoPE · LongVideoBench | score: 54.11 -> 52.68; 55.09 -> 52.73; 55.34 -> 53.01; 51.22 -> 46.32 | Hurt |
| Use larger test-time temporal scaling factors (ablation) | HoPE · LongVideoBench 32k | score: 54.36 (0.50) -> 55.70 (1.25) | Helped |
| Use smaller test-time temporal scaling factors (ablation) | HoPE · V-NIAH | accuracy: 61.78 (1.50) -> 63.56 (0.75) | Helped |
| Add hybrid frequency allocation (HFA) to the 3D positional structure (ablation) | HoPE · Video-MME, all reported context lengths | score: average improvement of 1.69 | Helped |
| Add dynamic temporal scaling (DTS) (ablation) | HoPE · Video-MME, 8k to 64k | score | Helped |
| Use a 3D positional structure instead of vanilla RoPE's flattened 1D positions (ablation) | Vanilla RoPE · Multimodal video tasks | performance | Helped |
Surprising findings
- Setting the lowest temporal frequencies to zero (NoPE-like) outperforms keeping all frequencies, contrary to the RoPE-everywhere default.
- Long video retrieval prefers smaller test-time scaling factors while long video understanding prefers larger ones.
Limitations
- Experiments use only 2B and 7B backbones and a subset of ~300k training pairs, with no larger models or data.
- Performance degrades sharply at 64k context on Video-MME, so extrapolation to extreme lengths remains hard.
- The semantic-preference theory relies on simplifying assumptions (i.i.d. query components, zero-mean perturbations, spatial terms assumed nonnegative).
Experimental setup
Datasets: LLaVA-Video-178k
Benchmarks: LongVideoBench Video-MME MLVU V-NIAH
Metrics: accuracy
Baselines: Vanilla RoPE M-RoPE VideoRoPE
Builds on: RoPE M-RoPE VideoRoPE NoPE
Closest papers by meaning
- Towards Long-window Anchoring in Vision-Language Model Distillation (AAAI 2026 · similarity 0.92)
- Revisiting Multimodal Positional Encoding in Vision–Language Models (ICLR 2026 · similarity 0.92)
- MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly (NeurIPS 2025 · similarity 0.89)
- Base of RoPE Bounds Context Length (NeurIPS 2024 · similarity 0.89)
- Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- Wavelet-based Positional Representation for Long Context (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.