Research map / Video / Video-language understanding
MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MA-LMM, which processes video frames online and stores past features in a visual and query memory bank with similarity-based compression, letting a BLIP-2-style multimodal LLM model long videos efficiently.
Problem
LLM-based large multimodal models can only process a limited number of video frames due to LLM context length and GPU memory limits, hindering long-term video understanding.
Approach
Process frames sequentially and store past visual features and learned queries in memory banks that serve as keys/values in the Q-Former's attention, with a memory bank compression that merges the most similar adjacent frame features so the bank length stays constant while keeping temporal order.
Key results
- Achieves 63.0 average top-1 on LVU, improving over the prior best S5 (59.2) by 3.8%.
- Reaches 93.0% on Breakfast and 93.2% on COIN, outperforming S5 by 2.3% and 2.4%.
- Gets 48.5 on MSRVTT-QA, 60.6 on MSVD-QA, and 49.8 on ActivityNet-QA, beating Video-LLaMA on all three.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add visual and query memory banks instead of using neither (ablation) | MA-LMM · LVU | top-1 accuracy: 48.3 -> 63.0 | Helped |
| Add the visual memory bank while retaining the query memory bank (ablation) | MA-LMM · LVU | top-1 accuracy: 58.0 -> 63.0 | Helped |
| Add the query memory bank while retaining the visual memory bank (ablation) | MA-LMM · LVU | top-1 accuracy: 61.5 -> 63.0 | Helped |
| Add both memory banks rather than neither (ablation) | MA-LMM · Breakfast | top-1 accuracy: 74.6 -> 93.0 | Helped |
| Add both memory banks rather than neither (ablation) | MA-LMM · COIN | top-1 accuracy: 72.3 -> 93.2 | Helped |
| Add a long-term memory bank in the off-the-shelf setting (ablation) | baseline large multimodal model · ActivityNet-QA | score: 29.9 -> 37.2 | Helped |
| Add a long-term memory bank in the off-the-shelf setting (ablation) | baseline large multimodal model · LVU | score: 23.6 -> 32.8 | Helped |
| Use memory bank compression (MBC) instead of FIFO memory eviction (ablation) | MA-LMM · LVU | top-1 accuracy: 61.3 -> 63.0 | Helped |
| Use memory bank compression (MBC) instead of FIFO memory eviction (ablation) | MA-LMM · Breakfast | top-1 accuracy: 88.5 -> 93.0 | Helped |
| Use memory bank compression (MBC) instead of FIFO memory eviction (ablation) | MA-LMM · COIN | top-1 accuracy: 90.4 -> 93.2 | Helped |
| Use Vicuna-7B rather than FlanT5-XL (ablation) | MA-LMM · MSRVTT-QA | top-1 accuracy: 46.5 -> 48.5 | Helped |
| Use Vicuna-7B rather than FlanT5-XL (ablation) | MA-LMM · MSVD-QA | top-1 accuracy: 57.6 -> 60.6 | Helped |
Surprising findings
- Adding the memory bank to the off-the-shelf InstructBLIP baseline without retraining improves results, e.g., +9.2 on LVU.
- Performance saturates when memory bank length is around 10 to 20 frames, suggesting high temporal redundancy in long videos.
Limitations
- Video QA on ActivityNet trails VideoCoCa, which uses large-scale video-text pretraining while MA-LMM uses only image-text pretraining.
- LVU evaluation limited to seven classification tasks since the LLM generates text and lacks regression capability.
Experimental setup
Datasets: LVU Breakfast COIN MSRVTT-QA MSVD-QA ActivityNet-QA MSRVTT MSVD YouCook2 EpicKitchens-100
Benchmarks: LVU Breakfast COIN MSRVTT-QA MSVD-QA ActivityNet-QA MSRVTT MSVD YouCook2 EpicKitchens-100
Metrics: top-1 accuracy METEOR CIDEr top-5 accuracy recall
Baselines: Video-LLaMA S5 ViS4mer VideoCoCa mPLUG-2 GIT Video-ChatGPT FIFO ToMe
Builds on: BLIP-2 InstructBLIP Video-LLaMA Vicuna
Closest papers by meaning
- Understanding Long Videos with Multimodal Language Models (ICLR 2025 · similarity 0.92)
- AdaCM^2: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reduction (CVPR 2025 · similarity 0.92)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.92)
- Leveraging Textual Memory and Key Frame Reasoning for Full Video Understanding Using Off-the-Shelf LLMs and VLMs (Student Abstract) (AAAI 2025 · similarity 0.91)
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.91)
- TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.