Research map / Vision-language and multimodal / Multimodal LLMs

Hybrid-Level Instruction Injection for Video Token Compression in Multi-modal Large Language Models

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HICom, an instruction-conditioned video token compressor for MLLMs that injects the user instruction at local (grouped) and global (learnable query) levels, plus a conditional pre-training stage on a new 248K-video dataset.

Problem

Video MLLMs incur high computational cost from many frame tokens, but existing compression (average pooling, Q-Former/Resampler, clustering, memory) is unconditional and discards instruction-relevant information.

Approach

Make token compression instruction-conditional at two levels: grouped local attention compresses each spatio-temporal group into one token, and global learnable queries attend over all frame tokens, with the instruction text injected into both via direct/adaptive-norm injection; a new conditional pre-training stage with instruction-followed descriptions aligns the injection modules.

Key results

• HICom-7B with 680 tokens (32 frames) and 1328 tokens (64 frames) gets +2.43% average over LLaVA-Video-7B on three multiple-choice benchmarks while saving 78.8% of visual tokens.
• Conditional compression with local+global (680 tokens) beats average pooling with 2592 tokens on all ablation benchmarks, reducing visual tokens by 73.8% and improving average by 0.47%.
• The conditional pre-training stage improves average performance by 1.17% for conditional compression in ablation.

Measured effects reported in the paper

Surprising findings

• Conditional compression at 680 tokens matches or beats unconditional average pooling at 2592 tokens, indicating that instruction guidance matters more than raw token count.
• Pre-training the injection modules on instruction-followed descriptions helps, but using multiple-choice QA data for this stage sharply degrades performance.

Limitations

• Fixed frame sampling during training limits long-video understanding, as acknowledged by the authors.
• Gains on VideoMME short videos are slightly below LLaVA-Video, which uses extra image data and unfrozen vision encoder.
• Global-level compression tends to forget fine-grained details; the approach is only explored on video, not images.

Experimental setup

Datasets: LLaVA-558K HICom-248K LLaVA-Video VideoChat2-IT M4-Instruct-Data Charades NTU RGB+D TVQA MiT Panda-70M Ego4D

Benchmarks: VideoMME MVBench EgoSchema ActivityNet VideoChatGPT Bench MLVU

Metrics: accuracy score

Baselines: LLaVA-Video LLaVA-OneVision VideoLLaMA2 LongVA Chat-Univi LLaMA-VID Tarsier VITA

Builds on: LLaVA-OneVision Q-Former LLaVA-Video Qwen2.5

Closest papers by meaning

• Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspective (ICLR 2026 · similarity 0.90)
• One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.89)
• DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models (CVPR 2025 · similarity 0.88)
• Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Prior (NeurIPS 2025 · similarity 0.88)
• Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs (NeurIPS 2025 · similarity 0.87)
• VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models (NeurIPS 2025 · similarity 0.87)