Research map / Video / Video-language understanding

AdaCodec: A Predictive Visual Code for Video MLLMs

NeurIPS 2026 · Video › Video-language understanding · approach: Video LLMs with fine-grained understanding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AdaCodec, a predictive visual code for video MLLMs that sends full ViT tokens only for adaptively placed reference (I) frames and compact motion-and-residual P-tokens for predictable frames, built on Qwen3-VL-8B.

Problem

Video MLLMs encode each sampled frame as an independent RGB image, so visual tokens repeat content already present in earlier frames, inflating context length and latency.

Approach

Replace the per-frame RGB visual interface with a predictive code: a codec-style GOP structure where I-frames are inserted by a pcost threshold on motion-search prediction cost, and P-frames are encoded as 5-channel (residual + motion vector) inputs tokenized by a widened ViT with learnable latent tokens, trained via teacher-feature alignment and then MLLM alignment.

Key results

• At 1/7 the visual-token budget (about 8.55k vs 55.9k tokens on general video), AdaCodec reaches 75.7 five-benchmark average vs 74.0 for Qwen3-VL-8B, with TTFT cut from 9.26s to 1.62s.
• At a matched 224k budget, AdaCodec improves MLVU by +3.1, LongVideoBench by +5.4 and TOMATO by +4.3 over the Qwen3-VL-8B baseline.
• AdaCodec at 32k visual tokens surpasses the 224k-token RGB baseline on all long-video benchmarks.

Surprising findings

• Thumbnail P-frames trained with the same budget underperform AdaCodec P-tokens by 2.4 to 3.8 points, indicating the motion-residual representation matters beyond the adaptive budget.
• Dynamic H.264-style macroblock partitioning, which the paper treats as the codec default, hurts accuracy by 1.4 to 1.9 points versus ViT-aligned 16x16 blocks.

Limitations

• Uses a fixed input resolution; dynamic resolution not explored.
• Uniform per-P-frame token budget (N_P=16) rather than adaptive to motion complexity.
• Not evaluated on streaming video.

Experimental setup

Datasets: ActivityNet-QA LongVILA TextVR Charades LSMDC TGIF-QA CharadesEgo Mementos TVQA CLEVRER Molmo2 Video-ChatGPT

Benchmarks: MLVU LongVideoBench LVBench TempCompass MotionBench TOMATO Video-MME MVBench NExT-QA PerceptionTest EgoSchema

Metrics: accuracy time-to-first-token end-to-end latency visual token count peak memory

Baselines: Qwen3-VL-8B InternVL3.5-8B Keye-VL-1.5-8B GLM-4.1V-9B MiniCPM-V-4.5-8B Eagle2.5-8B PLM-8B LLaVA-Video-7B VideoChat-Flash-7B Molmo2-8B CoPE-VideoLM-7B ReMoRa-7B

Builds on: Qwen3-VL H.264 predictive coding CoViAR lmms-eval YaRN

Closest papers by meaning

• MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.88)
• CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modelling (NeurIPS 2026 · similarity 0.87)
• HIMMEL: Hierarchical Interleaved Multi-stream Motion Encoding for Long Video Understanding (NeurIPS 2026 · similarity 0.86)
• Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.86)
• TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning (ICLR 2025 · similarity 0.86)
• Adaptive Keyframe Sampling for Long Video Understanding (CVPR 2025 · similarity 0.86)