Research map / Video / Video-language understanding
BIMBA: Selective-Scan Compression for Long-Range Video Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BIMBA, an MLLM that uses a Mamba-based selective-scan spatiotemporal token selector to compress long video token sequences (16x) before an LLM for long-form video question answering.
Problem
Long-form video question answering is hard because self-attention over massive spatiotemporal token sequences is prohibitively expensive, and existing compression (pooling, Perceiver, convolution) loses salient or long-range information.
Approach
A selective-scan (Mamba/S6) token selector that appends learned, average-pooled visual queries interleaved among spatiotemporal tokens and runs a bidirectional scan, compressing tokens by 16x while modeling long-range dependencies with linear cost.
Key results
- With Vicuna-7B on a shared 370K instruction set, BIMBA-LLaVA averages 55.23% vs 49.42% for LLaVA-NeXT (Video) and 50.40% for PLLaVA.
- BIMBA-LLaVA with Qwen2-7B (1.6M videos) reaches 83.73% on NExT-QA and 71.14% on EgoSchema, surpassing LongVU by 3.54% on EgoSchema.
- BIMBA-LLaVA (Qwen2-7B) outperforms LLaVA-Video by 7.11% on VNBench and sets new SOTA on LongVideoBench, Video-MME, and MLVU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace LLaVA-NeXT (Video) token processing with BIMBA selective-scan compression | Vicuna-7B video MLLM · NExT-QA | accuracy: 67.66 -> 72.35 | Helped |
| Replace PLLaVA token processing with BIMBA selective-scan compression | Vicuna-7B video MLLM · EgoSchema | accuracy: 43.36 -> 52.31 | Helped |
| Use BIMBA-LLaVA instead of LLaVA-Video | Qwen2-7B video MLLM · VNBench | accuracy: 70.77 -> 77.88 | Helped |
| Use BIMBA-LLaVA instead of LongVU | Qwen2-7B video MLLM · EgoSchema | accuracy: 67.60 -> 71.14 | Helped |
| Initialize visual queries with average pooling rather than learnable queries (ablation) | BIMBA-LLaVA · NExT-QA | accuracy: 67.67 -> 68.61 | Helped |
| Initialize visual queries with average pooling rather than learnable queries (ablation) | BIMBA-LLaMA · NExT-QA | accuracy: 68.91 -> 70.56 | Helped |
| Add layer normalization before token selection (ablation) | BIMBA-LLaVA · NExT-QA | accuracy: 68.61 -> 69.85 | Helped |
| Add layer normalization before token selection (ablation) | BIMBA-LLaMA · NExT-QA | accuracy: 70.56 -> 71.16 | Helped |
| Use bidirectional rather than standard selective scan (ablation) | BIMBA-LLaVA · NExT-QA | accuracy: 69.85 -> 71.68 | Helped |
| Use bidirectional rather than standard selective scan (ablation) | BIMBA-LLaMA · NExT-QA | accuracy: 71.16 -> 73.23 | Helped |
| Interleave queries among spatiotemporal tokens rather than append them (ablation) | BIMBA-LLaVA · NExT-QA | accuracy: 71.68 -> 73.57 | Helped |
| Interleave queries among spatiotemporal tokens rather than append them (ablation) | BIMBA-LLaMA · NExT-QA | accuracy: 73.23 -> 75.57 | Helped |
Surprising findings
- Naive spatiotemporal pooling saturates and even declines beyond 16 frames, while self-attention runs out of GPU memory beyond moderate lengths.
Limitations
- Question-conditioned token selection, which gives gains of about 1-2 points, is not used in the main model, so the default design does not exploit the query.
- Fair comparisons are restricted to models fine-tuned on the authors' 370K dataset, and the strongest results depend on a larger 1.6M-sample Qwen2-7B training run.
Experimental setup
Datasets: YouCook2 Ego4D-HCap NExT-QA IntentQA CLEVRER Ego4D STAR Perception Test
Benchmarks: PerceptionTest NExT-QA EgoSchema VNBench LongVideoBench Video-MME MLVU
Metrics: accuracy
Baselines: LLaVA-NeXT (Video) PLLaVA LLaMA-3.2 (Video) LLaVA-Video LongVU VideoTree Perceiver Self-Attention Pooling Vanilla
Builds on: Mamba LLaVA-NeXT LLaMA-3.2 Perceiver Vim (bidirectional selective scan)
Closest papers by meaning
- Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025 · similarity 0.89)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.88)
- Koala: Key Frame-Conditioned Long Video-LLM (CVPR 2024 · similarity 0.88)
- Searching Videos as Trees: Self-Correcting Agents for Grounded Long Video QA (NeurIPS 2026 · similarity 0.88)
- One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.88)
- Streaming Video Question-Answering with In-context Video KV-Cache Retrieval (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.