Research map / Video / Video-language understanding
AdaCM^2: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reduction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdaCM2, a cross-modality-attention-driven, layer-wise adaptive KV-cache reduction for the Q-Former that processes long videos frame-by-frame with bounded memory, evaluated on LVU, Breakfast, COIN, and captioning/VQA benchmarks.
Problem
Long-term video understanding with LLM-based models is memory-bound because visual tokens grow with frame count; prior compression methods use only visual similarity and ignore the text query.
Approach
Adaptively evict visual tokens from a frame-by-frame video cache in the Q-Former based on cross-modality (visual-to-text) attention scores, with per-layer split and conserve ratios, so memory converges to a constant as video length grows.
Key results
- Achieves 4.5% average accuracy gain on LVU over MA-LMM (e.g., 67.5 vs 63.0 average Top-1).
- Reports 94.4% on Breakfast and 93.3% on COIN, outperforming MA-LMM by 1.4 and 0.1 points.
- Reduces GPU memory by nearly 65% versus baselines while keeping memory nearly constant as frame count grows.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adaptive cross-modality memory reduction instead of MA-LMM's memory reduction | Long-term video understanding model · LVU | Average accuracy: +4.5% | Helped |
| Use adaptive cross-modality memory reduction instead of MA-LMM's memory reduction | Long-term video understanding model · Breakfast | Top-1 accuracy: 1.4% | Helped |
| Use adaptive cross-modality memory reduction instead of MA-LMM's memory reduction | Long-term video understanding model · COIN | Top-1 accuracy: 0.1% | Helped |
| Use adaptive cross-modality memory reduction instead of MA-LMM's memory reduction | AdaCM2 · MSVD | CIDEr: 10.3% | Helped |
| Use adaptive cross-modality memory reduction instead of MA-LMM's memory reduction | AdaCM2 · YouCook2 | CIDEr: 17.6 vs 17.6 | Hurt |
| Replace random eviction with cross-modality attention-based eviction (ablation) | AdaCM2 memory reduction · LVU | Top-1 accuracy | Helped |
| Replace Vicuna-7B with FlanT5-XL (ablation) | AdaCM2 · MSRVTT | METEOR: 33.0 -> 21.6 | Hurt |
| Replace Vicuna-7B with FlanT5-XL (ablation) | AdaCM2 · MSRVTT | CIDEr: 73.1 -> 58.9 | Hurt |
| Replace Vicuna-7B with FlanT5-XL (ablation) | AdaCM2 · MSVD | METEOR: 51.4 -> 48.7 | Hurt |
| Replace Vicuna-7B with FlanT5-XL (ablation) | AdaCM2 · MSVD | CIDEr: 189.4 -> 166.9 | Hurt |
| Replace Vicuna-7B with FlanT5-XL (ablation) | AdaCM2 · YouCook2 | METEOR: 17.6 -> 15.3 | Hurt |
| Replace Vicuna-7B with FlanT5-XL (ablation) | AdaCM2 · YouCook2 | CIDEr: 125.6 -> 107.6 | Hurt |
Surprising findings
- Cross-attention scores between adjacent frames exceed 90% cosine similarity, and similarity is higher in deeper layers.
Limitations
- Most memory is consumed by the LLM, so the reduction only affects the Q-Former portion unless a lighter LLM is used.
- Evaluated mainly on classification, captioning, and short-video QA; extremely long VQA benchmarks are limited.
- Only Vicuna-7B and FlanT5-XL compared for the LLM decoder; other LLMs left for future work.
Experimental setup
Datasets: LVU Breakfast COIN MSRVTT-QA MSVD-QA MSRVTT MSVD YouCook2
Benchmarks: LVU Breakfast COIN MSRVTT-QA MSVD-QA MSRVTT MSVD YouCook2
Metrics: Top-1 accuracy METEOR CIDEr GPU memory (GB)
Baselines: MA-LMM VideoLLaMA InstructBLIP Object transformer S4 ViS4mer mPLUG-2 GIT VideoCoca SwinBERT UniVL FlanT5-XL
Builds on: BLIP-2 InstructBLIP H2O MA-LMM
Closest papers by meaning
- MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding (CVPR 2024 · similarity 0.92)
- LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (ICML 2025 · similarity 0.90)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.90)
- ReWind: Understanding Long Videos with Instructed Learnable Memory (CVPR 2025 · similarity 0.90)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.89)
- $\infty$-Video: A Training-Free Approach to Long Video Understanding via Continuous-Time Memory Consolidation (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.