Research map / Video / Video-language understanding
BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BT-Adapter, a frozen-CLIP branching temporal adapter pretrained on WebVid-2M with asymmetric tube masking, enabling video conversation by plugging into image-language models without video instruction tuning.
Problem
Adapting image-language pretrained models and dialogue models (CLIP, LLaVA-style) to video efficiently, with strong temporal modeling and low GPU memory, and without costly video instruction data.
Approach
A branching temporal adapter runs alongside a frozen CLIP backbone, fusing backbone and branch features via learnable sigmoid selection; combined with asymmetric tube masking applied only in the branch and two alignment objectives (MBTA, MBCA), it is pretrained once and plugged into any image dialogue model.
Key results
- Zero-shot MSR-VTT text-to-video retrieval R@1 of 40.9 with BT-Adapter pretrained on 2M videos, versus 38.2 for TVTSv2 and 33.3 for UMT-L.
- BT-Adapter-LLaVA zero-shot video conversation mean score 2.46 vs 2.38 for fine-tuned VideoChatGPT; fine-tuned version reaches 2.69.
- Zero-shot ActivityNet-QA accuracy 46.1 for BT-Adapter (ZS) vs 35.2 for VideoChatGPT; pretraining takes 3 hours on 8 V100 GPUs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BT-Adapter-LLaVA without video instruction tuning instead of VideoChatGPT | Video conversation · VideoChatGPT benchmark | Mean Score: 2.38 -> 2.46 | Helped |
| Add video instruction tuning to BT-Adapter-LLaVA | LLaVA-Vicuna · VideoChatGPT benchmark | Mean Score: 2.46 -> 2.69 | Helped |
| Use BT-Adapter without video instruction tuning instead of VideoChatGPT | Video conversation · ActivityNet-QA | Acc: 35.2 -> 46.1 | Helped |
| Add video instruction tuning to BT-Adapter | LLaVA-Vicuna · MSRVTT-QA | Acc: 51.2 -> 57.0 | Helped |
| Use BT-Adapter instead of InternVideo | Zero-shot video-text retrieval · MSR-VTT | R@1: 40.7 -> 40.9 | Helped |
| Use BT-Adapter instead of Singularity | Zero-shot video-text retrieval · DiDeMo | R@1: 37.1 -> 35.6 | Hurt |
| Use BT-Adapter instead of UMT-L | Zero-shot video-text retrieval · LSMDC | R@1: 20.0 -> 19.5 | Hurt |
| Use BT-Adapter instead of VideoCoCa | Zero-shot video-text retrieval · ActivityNet | R@1: 34.5 -> 37.0 | Helped |
| Add 4-layer separate-ST modeling to CLIP (ablation) | BT-Adapter structure ablation · MSRVTT | R@1: 35.4 -> 35.7 | Helped |
| Replace separate-ST modeling with branch modeling (ablation) | BT-Adapter structure ablation · MSRVTT | R@1: 35.7 -> 37.4 | Helped |
| Add backbone-branch interaction (ablation) | BT-Adapter structure ablation · MSRVTT | R@1: 37.4 -> 38.5 | Helped |
| Add MBTA (ablation) | BT-Adapter pretraining · MSRVTT | R@1: 38.5 -> 39.3 | Helped |
Surprising findings
- The zero-shot BT-Adapter (no video instruction tuning) outperforms fine-tuned video chatbots such as VideoChatGPT on average in video conversation benchmarks.
Limitations
- Video instruction tuning still gives the best results; zero-shot adaptation trails on some benchmarks such as ActivityNet-based retrieval at R@1 on DiDeMo and LSMDC.
- Evaluation of video conversation relies on GPT-3.5 judged scores, which may be noisy.
Experimental setup
Datasets: WebVid-2M VideoChatGPT-100K MSR-VTT DiDeMo LSMDC ActivityNet Kinetics-400 HMDB-51 UCF-101 MSVD-QA MSRVTT-QA ActivityNet-QA
Benchmarks: MSR-VTT DiDeMo LSMDC ActivityNet Kinetics-400 HMDB-51 UCF-101 VideoChatGPT benchmark MSVD-QA MSRVTT-QA ActivityNet-QA
Metrics: R@1 R@5 R@10 Top-K accuracy Correctness of Information Temporal Understanding Consistency Detail Orientation Contextual Understanding QA accuracy
Baselines: VideoLLaMA LLaMA-Adapter VideoChat VideoChatGPT UMT Singularity InternVideo TVTSv2 VideoCoCa MiniGPT4 InstructBLIP
Builds on: CLIP LLaVA ST-Adapter VideoChatGPT divided space-time attention
Closest papers by meaning
- VideoLLM-online: Online Video Large Language Model for Streaming Video (CVPR 2024 · similarity 0.87)
- SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models (NeurIPS 2025 · similarity 0.87)
- VideoGLaMM : A Large Multimodal Model for Pixel-Level Visual Grounding in Videos (CVPR 2025 · similarity 0.87)
- Video-Panda: Parameter-efficient Alignment for Encoder-free Video-Language Models (CVPR 2025 · similarity 0.86)
- MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs (NeurIPS 2024 · similarity 0.86)
- Vision-Speech Models: Teaching Speech Models to Converse about Images (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.