Research map / Video / Video-language understanding
Slot-VLM: Object-Event Slots for Video-Language Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Slot-VLM replaces pooling/Q-Former video token compression with a dual-branch Object-Event Slots module (slot attention over high-res sparse frames for objects and over low-res dense frames for events) feeding a frozen Vicuna-7B for video QA.
Problem
Video-language models produce too many visual tokens from dense frames, and existing compressors (pooling, Q-Former, clustering) yield semantically entangled tokens that align poorly with LLM word tokens.
Approach
Compress video features into a small set of semantically decoupled object-centric slots (spatial slot attention on high-resolution, low-frame-rate features) and event-centric slots (temporal slot attention per spatial position on high-frame-rate, low-resolution features), then project them as LLM inputs.
Key results
- Slot-VLM achieves 74.9% accuracy on MSVD-QA, 69.6% on MSRVTT-QA, and 48.3% on ActivityNet-QA with only 192 video tokens and 100K video pairs; the 665K-image variant averages 65.0% across three open-ended benchmarks.
- Replacing slot attention with Q-Former at matched FLOPs lowers in-domain accuracy (e.g., ST-QFormer-VLM 29.7% vs. Slot-VLM 48.8%) and MSVD-QA accuracy.
- Slot-VLM reaches 55.8% on the 500-question EgoSchema subset, above Video-LLaVA (36.8%) and LLoVi (50.8%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the Event-Slots branch to the Object-Slots branch (ablation) | Slot-VLM · In-domain | accuracy: 46.5 -> 48.8 | Helped |
| add the Event-Slots branch to the Object-Slots branch (ablation) | Slot-VLM · MSVD-QA | accuracy: 73.1 -> 74.9 | Helped |
| replace spatial Q-Former with slot attention (ablation) | spatial branch · In-domain | accuracy: 40.1 -> 46.5 | Helped |
| replace spatial Q-Former with slot attention (ablation) | spatial branch · MSVD-QA | accuracy: 70.9 -> 73.1 | Helped |
| replace temporal Q-Former with slot attention (ablation) | temporal branch · In-domain | accuracy: 46.0 -> 47.1 | Helped |
| replace temporal Q-Former with slot attention (ablation) | temporal branch · MSVD-QA | accuracy: 72.6 -> 73.1 | Helped |
| use two slot-attention branches rather than two Q-Former branches (ablation) | video QA model · In-domain | accuracy: 29.7 -> 48.8 | Helped |
| use two slot-attention branches rather than two Q-Former branches (ablation) | video QA model · MSVD-QA | accuracy: 66.9 -> 74.9 | Helped |
| use two-branch slot learning instead of jointly learning slots from dense video features (ablation) | Slot-VLM · In-domain | accuracy: 46.7 -> 48.8 | Helped |
| use two-branch slot learning instead of jointly learning slots from dense video features (ablation) | Slot-VLM · MSVD-QA | accuracy: 72.8 -> 74.9 | Helped |
| retain high spatial resolution in the Object-Slots branch (ablation) | Object-Slot-VLM · In-domain | accuracy: 41.9 -> 46.5 | Helped |
| retain high spatial resolution in the Object-Slots branch (ablation) | Object-Slot-VLM · MSVD-QA | accuracy: 69.8 -> 73.1 | Helped |
Surprising findings
- Adding the Q-Former-based two-branch variant (ST-QFormer-VLM) performs worse than either single-branch Q-Former variant, suggesting alignment difficulty when combining token sets.
- Instruction tuning appears to make slot attention masks more decoupled compared with stage-1 reconstruction-only pretraining.
Limitations
- Learned slots are imperfect: object instances are not perfectly segmented and events are not cleanly partitioned.
- Not validated on very long (hour-scale) videos; the authors propose chunking as future work.
- Compared methods use differing pretraining and instruction data, so comparisons are not fully controlled.
Experimental setup
Datasets: Video Instruction Data (Video-ChatGPT, 100K video QA pairs from ActivityNet) LLaVA v1.5 665K image-text instruction data MSVD-QA MSRVTT-QA ActivityNet-QA EgoSchema NExT-QA STAR
Benchmarks: MSVD-QA MSRVTT-QA ActivityNet-QA EgoSchema NExT-QA STAR
Metrics: accuracy average score (0-5)
Baselines: Video-LLaMA VideoChat Video-ChatGPT Chat-UniVi Video-LLaVA BT-Adapter LLaMA-VID VideoChat2 MovieChat Q-Former variants (Spatial/Temporal/ST-QFormer-VLM)
Builds on: Slot Attention DINOSAUR SLATE SlowFast Video-ChatGPT BLIP-2
Closest papers by meaning
- Slot State Space Models (NeurIPS 2024 · similarity 0.88)
- Leveraging Textual Memory and Key Frame Reasoning for Full Video Understanding Using Off-the-Shelf LLMs and VLMs (Student Abstract) (AAAI 2025 · similarity 0.86)
- CTRL-O: Language-Controllable Object-Centric Visual Representation Learning (CVPR 2025 · similarity 0.86)
- Parallelized Spatiotemporal Slot Binding for Videos (ICML 2024 · similarity 0.86)
- MetaSlot: Break Through the Fixed Number of Slots in Object-Centric Learning (NeurIPS 2025 · similarity 0.86)
- VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLM (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.