Research map / Video / Video-language understanding
TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TOPA, which uses LLM-generated textual videos (TextVid, 721K) and CLIP text features to pre-align LLMs with video, then applies them to real videos zero-shot via CLIP feature projection or fine-tunes them.
Problem
Extending LLMs to video understanding without costly real video-text pre-training, given noisy web video-text supervision and the difficulty of long-form video understanding (e.g., EgoSchema).
Approach
Generate "textual videos" (sequences of textual frames with dense descriptions and QA) with an LLM, encode frames with the CLIP text encoder, and train a projection plus Llama-adapter on these CLIP text feature sequences; at inference, project real CLIP image features into the text feature space via a memory-based training-free mapping (DeCap-style).
Key results
- TOPA-Llama2-13B achieves 51.0% Top-1 on EgoSchema full set zero-shot without real video training, above prior video-text pretraining methods and competitive with GPT-3.5/GPT-4 video agents.
- TOPA-Llama2-7B reaches 67.1% on TVQA after fine-tuning, versus 63.8% for the baseline without pre-alignment.
- Zero-shot video captioning CIDEr of 32.9 on MSR-VTT and 31.0 on VATEX for TOPA-Llama2-7B, above DeCap (18.6/18.7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TOPA text-only pre-alignment | Llama2-13B · EgoSchema full set | Acc@1: 41.2 -> 51.0 | Helped |
| Add TOPA text-only pre-alignment | Llama2-7B · TVQA validation after supervised fine-tuning | Accuracy: 63.8 -> 67.1 | Helped |
| Use CLIP modality projection at inference (ablation) | TOPA-Llama2-13B · EgoSchema full set | Acc@1: 38.3 -> 51.0 | Helped |
| Use CLIP modality projection at inference (ablation) | TOPA-Llama2-7B · EgoSchema full set | Acc@1: 30.6 -> 41.2 | Helped |
| Increase the input from 1 frame to 10 frames (ablation) | TOPA-Llama2-13B · EgoSchema full set | Acc@1: 47.6 -> 51.0 | Helped |
| Increase the input from 1 frame to 10 frames (ablation) | TOPA-Llama2-7B · EgoSchema full set | Acc@1: 39.4 -> 41.2 | Helped |
| Add multi-choice Tideo QA pre-training (ablation) | TOPA-Llama2-7B · EgoSchema full set, LLM selection | Accuracy: 24.7 -> 41.2 | Helped |
| Add multi-choice Tideo QA pre-training (ablation) | TOPA-Llama2-7B · EgoSchema full set, LLM logits | Accuracy: 40.5 -> 41.7 | No clear effect |
Surprising findings
- Removing the modality projection drops TOPA-Llama2-13B EgoSchema accuracy from 51.0% to 38.3%, showing the CLIP modality gap is critical.
- TOPA underperforms SeViLA and IG-VLM on NExT-QA and STAR, which the authors attribute to fine-grained visual details being lost in text-feature projection.
Limitations
- Modality gap between CLIP text and image features degrades performance and limits fine-grained visual understanding (object locations, relations).
- Focus on keyframes makes it unsuitable for high-fps or many-frame tasks such as action counting.
Experimental setup
Datasets: TextVid NExT-QA STAR TVQA WebVid-2M Howto100M Ego4D WordNet CC3M
Benchmarks: EgoSchema NExT-QA STAR TVQA MVBench MSR-VTT VATEX
Metrics: Top-1 accuracy CIDEr
Baselines: FrozenBiLM InternVideo LongViViT MC-ViT-L SeViLA IG-VLM Vamos LLoVi VideoAgent DeCap VideoCoCa VideoChat2
Builds on: CLIP LLaMA-Adapter DeCap Flipped-VQA
Closest papers by meaning
- VidLA: Video-Language Alignment at Scale (CVPR 2024 · similarity 0.90)
- VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models (CVPR 2025 · similarity 0.90)
- Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval (AAAI 2024 · similarity 0.89)
- SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understanding (CVPR 2025 · similarity 0.89)
- Leveraging Textual Memory and Key Frame Reasoning for Full Video Understanding Using Off-the-Shelf LLMs and VLMs (Student Abstract) (AAAI 2025 · similarity 0.89)
- Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.