Research map / Video / Video-language understanding
SEAL: Semantic Attention Learning for Long Video Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SEAL, a unified long-video representation that decomposes videos into scene, object, and action tokens and selects a query-relevant, diverse subset via attention learning, used with MLLM or vision heads for video QA and temporal grounding.
Problem
Long video understanding is hard due to high computational cost, temporal redundancy, and the need for representations that generalize across tasks such as video QA and temporal grounding.
Approach
Decompose a long video into scene, object (SAM masks), and action (tracklet) tokens, then select a fixed-size subset by maximizing query relevance (BLIP-2 similarity) plus token diversity (pairwise dissimilarity), with a global mode and a sliding-window streaming mode.
Key results
- On LVBench SEAL reaches 45.9% overall accuracy, above Qwen2-VL-72B at 41.3%, with 34B LLM.
- On MovieChat-1K SEAL reaches 86.8% accuracy, 1.9 points above HERMES (84.9%).
- On Ego4D-NLQ under 200-token budget, SEAL improves R@1 average from 8.19 (SnAG) to 8.95.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SEAL instead of Qwen2-VL | LVBench · LVBench | overall accuracy: 41.3 -> 45.9 | Helped |
| use SEAL instead of HERMES | MovieChat-1K · MovieChat-1K | accuracy: 84.9 -> 86.8 | Helped |
| use SEAL instead of HERMES | MovieChat-1K · MovieChat-1K | score: 4.40 -> 4.35 | Hurt |
| use SEAL instead of SnAG with the same token budget | Ego4D-NLQ, 450 tokens · Ego4D-NLQ | R@1 Avg: 11.34 -> 11.52 | Helped |
| use SEAL instead of SnAG with the same token budget | Ego4D-NLQ, 200 tokens · Ego4D-NLQ | R@1 Avg: 8.19 -> 8.95 | Helped |
| add scene, object, and action tokens instead of random token inputs (ablation) | SEAL · LVBench | overall accuracy: 24.8 -> 45.9 | Helped |
| add scene tokens (ablation) | SEAL · LVBench | overall accuracy: 24.8 -> 42.8 | Helped |
| add action tokens to scene tokens (ablation) | SEAL · LVBench | overall accuracy: 42.8 -> 44.4 | Helped |
| add object tokens to scene and action tokens (ablation) | SEAL · LVBench | overall accuracy: 44.4 -> 45.9 | Helped |
| add object tokens to scene tokens (ablation) | SEAL · MovieChat-1K | accuracy: 76.33 -> 79.49 | Helped |
| add action tokens to scene tokens (ablation) | SEAL · MovieChat-1K | accuracy: 76.33 -> 81.46 | Helped |
| combine query relevance and token diversity in attention learning (ablation) | SEAL · LVBench | overall accuracy: 38.7 -> 45.9; 42.0 -> 45.9 | Helped |
Surprising findings
- Using only action tokens (34.0%) or object tokens (33.9%) alone already greatly outperforms random tokens (24.8%) on LVBench, and scene tokens alone reach 42.8%.
- Streaming mode performs worse than global mode on EU and KIR but better on TG and Rea.
Limitations
- Attention learning uses a QP solver bounded by memory, so the system is not fully end-to-end trainable.
- Weak on causal reasoning questions where MLLM heads show limitations.
Experimental setup
Datasets: LVBench MovieChat-1K Ego4D-NLQ
Benchmarks: LVBench MovieChat-1K Ego4D-NLQ
Metrics: accuracy score Recall@1 Recall@5
Baselines: Qwen2-VL-72B InternVL2 LLaVA-NeXT Oryx PLLaVA SnAG HERMES MovieChat TimeChat-Hal VideoChat
Builds on: SAM SAM 2 BLIP-2 SnAG InternVL2
Closest papers by meaning
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.90)
- One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.90)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.89)
- LongVPO: From Anchored Cues to Self-Reasoning for Long-Form Video Preference Optimization (NeurIPS 2025 · similarity 0.89)
- $\infty$-Video: A Training-Free Approach to Long Video Understanding via Continuous-Time Memory Consolidation (ICML 2025 · similarity 0.88)
- FlexSelect: Flexible Token Selection for Efficient Long Video Understanding (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.