Research map / Video / Video-language understanding
Koala: Key Frame-Conditioned Long Video-LLM
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Koala adapts a frozen short-video LLM (Video-LLaMA) to minutes-long videos by adding learnable spatiotemporal queries conditioned on sparse key-frame tokens, trained self-supervised on HowTo100M.
Problem
Pretrained short-clip video LLMs struggle to understand minutes-long videos and answer questions about long-term temporal structure.
Approach
Introduces Conditioned Segment and Conditioned Video tokenizers that repurpose the frozen video QFormer with learnable queries conditioned on global key-frame tokens, so segment-level and cross-segment tokens carry global context before entering the LLM.
Key results
- Koala reaches 40.42% top-1 on EgoSchema vs 33.25% for the base Video-LLaMA (zero-shot).
- On Seed-Bench procedure understanding Koala scores 35.91% vs 25.42% for Video-LLaMA-2.
- On Seed-Bench action recognition Koala scores 41.26% vs 35.52% for Video-LLaMA-2, outperforming VPGTrans (39.54%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the Conditioned Segment tokenizer (ablation) | Video-Llama · EgoSchema | Top 1 Acc (%): 33.25 -> 36.93 | Helped |
| add the Conditioned Segment tokenizer (ablation) | Video-Llama · Seed-Bench Procedure Understanding | Accuracy: 26.68 -> 30.20 | Helped |
| add the Conditioned Segment tokenizer (ablation) | Video-Llama · Seed-Bench Action Recognition | Accuracy: 35.52 -> 38.74 | Helped |
| add the Conditioned Video tokenizer after CS (ablation) | Video-Llama + CS · EgoSchema | Top 1 Acc (%): 36.93 -> 40.42 | Helped |
| add the Conditioned Video tokenizer after CS (ablation) | Video-Llama + CS · Seed-Bench Procedure Understanding | Accuracy: 30.20 -> 35.91 | Helped |
| add the Conditioned Video tokenizer after CS (ablation) | Video-Llama + CS · Seed-Bench Action Recognition | Accuracy: 38.74 -> 41.26 | Helped |
| retain the CS tokenizer output tokens for zkey (ablation) | Koala · EgoSchema | Top 1 Acc (%): 40.42 -> 33.61 | Hurt |
| remove zkey conditioning in the CS tokenizer (ablation) | Koala · EgoSchema | Top 1 Acc (%): 39.12 -> 40.42 | Hurt |
| remove temporal concept queries Qtemp and Qconcepts (ablation) | Koala · EgoSchema | Top 1 Acc (%): 39.20 -> 40.42 | Hurt |
| replace Koala temporal aggregation with average pooling (ablation) | Video-Llama · EgoSchema | Top 1 Acc (%): 33.39 vs 40.33 | Hurt |
| replace Koala temporal aggregation with a memory module (ablation) | Video-Llama · EgoSchema | Top 1 Acc (%): 34.62 vs 40.33 | Hurt |
| replace Koala temporal aggregation with segment-token concatenation (ablation) | Video-Llama · EgoSchema | Top 1 Acc (%): 35.72 vs 40.33 | Hurt |
Surprising findings
- Naively concatenating segment tokens or using a memory module does not clearly beat the proposed tokenizers, and a Flan-T5 language prior without video outperforms most video-language models on EgoSchema.
- Finetuning Video-LLaMA on HowTo100M without the CS and CV functions drops top-1 accuracy by about 5%.
Limitations
- Limited by the base vLLM's maximum input token count, which caps the number of segments.
- Extending positional embeddings to much longer videos such as movies is not addressed.
- Unclear generalization to much longer videos beyond minutes-long settings.
Experimental setup
Datasets: HowTo100M
Benchmarks: EgoSchema Seed-Bench
Metrics: top-1 accuracy
Baselines: Video-LLaMA Flan-T5 language prior MovieChat InternVideo VPGTrans mPLUG-Owl VideoChat Valley Video-ChatGPT InstructBLIP Frozen-BiLM VIOLET
Builds on: Video-LLaMA QFormer Perceiver SlowFast
Closest papers by meaning
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.90)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.89)
- Unleashing Hour-Scale Video Training for Long Video-Language Understanding (NeurIPS 2025 · similarity 0.89)
- Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025 · similarity 0.89)
- MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding (CVPR 2024 · similarity 0.89)
- Frame-Voyager: Learning to Query Frames for Video Large Language Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.