Research map / Video / Video-language understanding

CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modelling

NeurIPS 2026 · Video › Video-language understanding · approach: Long video understanding

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Encodes MPEG-4 motion vectors and residuals as compact Δ-tokens interleaved with I-frame RGB tokens to build a codec-aware token sequence for a 7B VideoLM, cutting TTFT by up to 86% and tokens by up to 93%.

Problem

Video language models are limited by context windows and keyframe sampling, which gives sparse temporal coverage and wastes computation on redundant full RGB frames, inflating prefill time and TTFT.

Approach

Replace RGB encoding of P-frames with a lightweight Δ-encoder over codec primitives (motion vectors and residuals), pre-trained to align with the image encoder's token space via patch-wise MSE regression, then interleaved temporally with I-frame tokens and fine-tuned end-to-end in LLaVA-Video-7B.

Key results

• Ours-7B reaches 70.3 on PerceptionTest and 60.3 on ActivityNet-QA, the best among open-source models in the paper's table, and 46.4 on LVBench vs 44.2 for LLaVA-Video-7B.
• Relative to 64-frame LLaVA-Video-7B, the 1-keyframe-per-GOP configuration cuts time-to-first-token by 86.2% and end-to-end latency by 56.1%.
• With 4 keyframes + 4 P-frames per GOP, the model gets 70.3 on PerceptionTest (+6.7 over matched LLaVA-Video at 63.6) using about 20% of the tokens.

Surprising findings

• Fine-tuning with codec primitives improves the model even when Δ-tokens are removed at inference (I-frame-only evaluation still beats LLaVA-Video at matched keyframe counts).
• On VideoMME the model trails LLaVA-Video-7B (61.9 vs 63.3 w/o subtitles), which the authors attribute to training data mixture rather than the codec formulation.

Limitations

• Only I- and P-frames are supported; B-frames with non-causal dependencies are not handled.
• Operates on tensorized codec primitives rather than raw block-wise motion vectors and DCT coefficients, and uses a fixed P-frame fusion window.
• Trails LLaVA-Video-7B on VideoMME, attributed to smaller training corpus.

Experimental setup

Datasets: LLaVA-Video-178K ActivityNet-QA NextQA PerceptionTest ScanQA SQA3D

Benchmarks: PerceptionTest NextQA ActivityNet-QA VideoMME TempCompass TOMATO CVRR-ES MVBench LongVideoBench LVBench Video-TT Video-MMMU

Metrics: accuracy exact match CIDEr TTFT end-to-end latency token usage

Baselines: LLaVA-Video-7B EMA-7B Video-LaVIT FastV DyCoke PLLaVA VisionZip LLaVA-Scissor LLaVA-OV-7B VILA-40B

Builds on: LLaVA-Video CoViAR EMA Video-LaVIT CompressedVideoMAE

Closest papers by meaning

• MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.88)
• AdaCodec: A Predictive Visual Code for Video MLLMs (NeurIPS 2026 · similarity 0.87)
• One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.87)
• Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.87)
• A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Frames (CVPR 2024 · similarity 0.87)
• Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.87)