Research map / Video / Video-language understanding
VLog: Video-Language Models by Generative Retrieval of Narration Vocabulary
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VLog, a generative-retrieval video-language model built on GPT-2 that retrieves narrations from a hierarchical, Ego4D-derived narration vocabulary (via narration pair encoding) instead of generating subword tokens, with an LMM+LLM agentic workflow to expand the vocabulary.
Problem
Generative video-language models decode token-by-token over large subword vocabularies, which is slow for dense narration of long videos and poorly suited to visually grounded outputs; pure retrieval models are fast but cannot handle reasoning queries.
Approach
A retrieval token appended to a GPT-2 sequence (with SigLIP visual features) produces an embedding scored by dot product against a fixed narration vocabulary, combining causal reasoning with fast retrieval; narrations are split into prefixes and postfixes by Narration Pair Encoding, indexed hierarchically by scene, and the vocabulary is expanded with LLaVA-OV and Qwen2.5 when OOV events are detect
Key results
- On VidCap-Eval causal retrieval, VLog reaches CIDEr 87.3 and R@1 5.0 versus 53.7 / 3.1 for the GPT2-32K generative model, with decoding 20x faster (0.018 s vs 0.362 s per clip).
- On EgoSchema, VLog with LLaMA3-8B answerer gets 70.4 subset accuracy and 43.1 full accuracy, improving over LLoVi (67.0 / 38.8) and running in 2.3 s per 180 s video.
- On COIN step recognition, VLog with Ego4D pretraining reaches 57.4% top-1 accuracy, versus 44.6% for the GPT2-medium generative baseline and 59.8% for VideoLLM-online-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VLog instead of a generative GPT-2 narrator | COIN next-step forecasting · COIN next-step forecasting | Top-1 Acc: 32.1 -> 48.4 | Helped |
| Use VLog as narrator instead of LLoVi | EgoSchema QA pipeline · EgoSchema subset | accuracy: 67.0 -> 70.4 | Helped |
| Add HiREST-specific vocabulary items to the COIN vocabulary (ablation) | VLog · HiREST step captioning | CIDEr: 6.9 -> 12.6 | Helped |
| Add postfix retrieval to the VLog prefix vocabulary (ablation) | VLog · Vidcab-Eval casual retrieval | CIDEr: 83.9 -> 86.9 | Helped |
| Use hierarchical indexing instead of brute-force search (ablation) | VLog · EgoSchema val | time (s): 34.5 -> 2.3 | Helped |
| Initialize the retrieval token by pooling visual features instead of using EOS (ablation) | VLog · COIN step | Top-1 Acc: 51.3 -> 54.0 | Helped |
| Use pretrained GPT2 weights rather than random initialization (ablation) | VLog · COIN step | Top-1 Acc: 51.5 -> 54.0 | Helped |
| Use a larger GPT2 model (ablation) | VLog · COIN step | Top-1 Acc: 54.0 -> 56.4 | Helped |
Surprising findings
- Using the large Ego4D narration vocabulary (0.8M) underperforms a small COIN-specific vocabulary on HiREST step captioning, suggesting domain mismatch matters more than vocabulary size.
Limitations
- Predefined vocabulary constrains outputs to known narrations and limits generalization to open-ended queries.
- Vocabulary expansion relies on a small LMM and LLM (0.5B each) that can hallucinate fine-grained actions.
- Results on egocentric data may not transfer well to other domains, as shown by weaker performance with the Ego4D vocabulary on HiREST.
Experimental setup
Datasets: Ego4D COIN EgoSchema HiREST
Benchmarks: VidCap-Eval COIN EgoSchema HiREST
Metrics: CIDEr Recall@1 decode time (s) top-1 accuracy accuracy METEOR SPICE
Baselines: GPT2-medium generative baseline SigLIP-L retrieval with mean-pooling SigLIP-L retrieval with adapter VideoLLM-online-7B LLoVi LangRepo MVU LLaVA-OV-0.5B
Builds on: GPT-2 SigLIP Ego4D narrations Byte Pair Encoding LoVi
Closest papers by meaning
- Leveraging Textual Memory and Key Frame Reasoning for Full Video Understanding Using Off-the-Shelf LLMs and VLMs (Student Abstract) (AAAI 2025 · similarity 0.88)
- VISTA-LLAMA: Reducing Hallucination in Video Language Models via Equal Distance to Visual Tokens (CVPR 2024 · similarity 0.87)
- Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understanding (NeurIPS 2025 · similarity 0.87)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.87)
- SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding (ICLR 2025 · similarity 0.86)
- HourVideo: 1-Hour Video-Language Understanding (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.