Research map / Video / Video-language understanding
VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoICL, a training-free video in-context learning framework that retrieves similar demonstrations and iteratively re-runs a video LMM on new example batches until token-probability confidence is high, improving OOD video understanding.
Problem
Video LMMs perform poorly on out-of-distribution video tasks underrepresented in training data, and in-context learning for video is limited because video tokens are long, allowing only a few demonstrations within the context window.
Approach
Select top-k demonstrations by a linear combination of text and video embedding cosine similarity, then feed them in small batches (m=2) iteratively, stopping when the minimum token probability of the generated answer exceeds a threshold and otherwise keeping the most confident answer.
Key results
- VideoICL with LLaVA-Video-7B improves over zero-shot by +4.3 on Animal Kingdom, +54.6 on PitVQA and +14.0 on UCF-Crime, and outperforms zero-shot LLaVA-Video-72B on Animal Kingdom (72.3 vs 69.7).
- On UCF-Crime, VideoICL (53.3) exceeds LoRA fine-tuned LLaVA-Video-7B (51.9).
- Token probability confidence outperforms verbalization and trained probe for confidence estimation (e.g., Animal Kingdom 72.3 vs 69.7 and 71.7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use similarity-ranked demonstrations and confidence-based iterative inference | LLaVA-Video-7B · Animal Kingdom | accuracy: 68.0 -> 72.3 (+4.3) | Helped |
| Use similarity-ranked demonstrations and confidence-based iterative inference | LLaVA-Video-7B · Sports-QA | accuracy: 25.5 -> 47.6 (+22.1) | Helped |
| Use similarity-ranked demonstrations and confidence-based iterative inference | LLaVA-Video-7B · PitVQA | accuracy: 6.7 -> 61.3 (+54.6) | Helped |
| Use similarity-ranked demonstrations and confidence-based iterative inference | LLaVA-Video-7B · UCF-Crime | accuracy: 39.3 -> 53.3 (+14.0) | Helped |
| Use similarity-ranked demonstrations and confidence-based iterative inference | LLaVA-Video-7B · Drive&Act | accuracy: 20.2 -> 53.4 (+33.2) | Helped |
| Select demonstrations randomly instead of by similarity (ablation) | VIDEOICL · Animal Kingdom | accuracy: 68.0 -> 68.4 (+0.4) | No clear effect |
| Select demonstrations randomly instead of by similarity (ablation) | VIDEOICL · PitVQA | accuracy: 6.7 -> 8.3 (+1.6) | No clear effect |
| Select demonstrations randomly instead of by similarity (ablation) | VIDEOICL · UCF-Crime | accuracy: 39.3 -> 38.4 (-0.9) | Hurt |
| Use text features only for demonstration selection (ablation) | VIDEOICL · PitVQA | accuracy: 61.3 -> 33.1 (+24.8) | Hurt |
| Use video features only for demonstration selection (ablation) | VIDEOICL · PitVQA | accuracy: 61.3 -> 29.1 (+22.4) | Hurt |
| Increase the demonstration pool from k = 2 to k = 8 (ablation) | VIDEOICL · PitVQA | accuracy: 54.0 -> 61.3 | Helped |
| Increase the demonstration pool from k = 8 to k = 16 (ablation) | VIDEOICL · PitVQA | accuracy: 61.3 -> 61.2 | No clear effect |
Surprising findings
- Only 29% of CapERA responses and 50% of PitVQA responses reach the confidence threshold in the first iteration despite using the most similar examples.
- Verbalized confidence performed worst, contrary to findings in prior work that trained probes are most accurate.
Limitations
- Iterative inference multiplies inference calls (up to n=k/m=4 passes per query), adding latency.
- Relies on confidence estimates from token probabilities, which may be unreliable for some tasks.
- Context-length limits restrict each iteration to m=2 examples.
Experimental setup
Datasets: Animal Kingdom Sports-QA PitVQA UCF-Crime Drive&Act CapERA
Benchmarks: Animal Kingdom Sports-QA PitVQA UCF-Crime Drive&Act CapERA
Metrics: accuracy BLEU-1 BLEU-2 BLEU-3 BLEU-4 METEOR ROUGE-L
Baselines: GPT-4o Gemini-1.5 Pro Otter-7B MMICES SimRankOnce RandExVote SimRankVote LLaVA-Video-72B zero-shot LoRA fine-tuned LLaVA-Video-7B
Builds on: in-context learning Iterative demonstration selection MMICES Otter
Closest papers by meaning
- Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- Leveraging Textual Memory and Key Frame Reasoning for Full Video Understanding Using Off-the-Shelf LLMs and VLMs (Student Abstract) (AAAI 2025 · similarity 0.88)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.88)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.88)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.88)
- Incentivizing Versatile Video Reasoning in MLLMs via Data-Efficient Reinforcement Learning (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.