Research map / Video / Video-language understanding

HIMMEL: Hierarchical Interleaved Multi-stream Motion Encoding for Long Video Understanding

NeurIPS 2026 · Video › Video-language understanding · approach: Long video understanding

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

HIMMEL routes sparse anchor frames to the host ViT and dense inter-frame intervals to a lightweight tri-stream compressed-domain adapter (motion vectors, residuals, I-frame context), aligned with InfoNCE and injected into the LLM, reaching 63.5% on Video-MME with 3.6x fewer visual tokens on Qwen2.5-VL-7B.

Problem

Long-video multimodal LLMs face decode cost, quadratic context growth from frame tokens, and weak motion perception under sparse keyframe sampling.

Approach

Allocate expensive host-ViT processing to sparse semantic anchor frames and encode dense inter-frame intervals with a lightweight tri-stream adapter over codec motion vectors, residuals, and I-frame context, aligned to host visual space via InfoNCE contrastive pre-alignment before LoRA-based SFT with differentiable placeholder injection.

Key results

• On Video-MME, HIMMEL (Qwen2.5-VL-7B) reaches 63.5% vs 61.2% for the dense 32-frame baseline while using 12.2k vs 44.7k visual context tokens (3.6x fewer).
• With Qwen3-VL-8B host, HIMMEL reaches 64.9% on Video-MME.
• Replacing InfoNCE with MSE alignment drops Video-MME from 63.5% to 62.0%; same-SFT dense host reaches 62.04%, so about 1.48 pp of the gain is over same-data SFT.

Surprising findings

• MV-only or residual-only tokens without I-frame context hurt performance below the 8-anchor-only configuration (e.g., MV+residual without I-frame context 50.8% vs 58.0% anchors-only).
• Video-only SFT slightly lowers MathVista and GSM8K retention relative to the untouched host (682 to 681 and 1137 to 1129 correct).

Limitations

• Single-video scope; no cross-video memory or retrieval.
• No audio or OCR branch; motion/residual inputs do not decode textual semantics.
• Reported Wilson CIs are item-level and do not capture training-seed variance; baseline comparisons are not all matched protocols.

Experimental setup

Datasets: LLaVA-Video-178K Video-MME MVBench MLVU MathVista GSM8K

Benchmarks: Video-MME MVBench MLVU MathVista-testmini GSM8K-test LongVideoBench PerceptionTest VSI-SUPER ActivityNet-QA

Metrics: accuracy Wilson 95% CI latency (s) samples/s peak GiB MRA

Baselines: Dense 32-frame baseline CoPE-VideoLM ReMoRa Video-LaVIT AKS QuoTA BOLT Static-or-Dynamic LOOK-M HERMES CodecSight LongVU

Builds on: CoViAR InfoNCE LoRA Qwen2.5-VL

Closest papers by meaning

• TTVidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining (NeurIPS 2026 · similarity 0.88)
• MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.88)
• Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video LLMs (NeurIPS 2026 · similarity 0.88)
• Improving LLM Video Understanding with 16 Frames Per Second (ICML 2025 · similarity 0.88)
• V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Models (NeurIPS 2026 · similarity 0.87)
• OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence (NeurIPS 2026 · similarity 0.87)