Research map / Video / Video-language understanding
Unifying Specialized Visual Encoders for Video Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MERV, a VideoLLM that fuses four heterogeneous frozen/fine-tuned visual encoders (DINOv2, ViViT, SigLIP, LanguageBind) via spatio-temporally aligned pre-fusion pooling and cross-attention mixing, with a LLaMA-2 7B LLM.
Problem
Video LLMs typically rely on a single vision encoder from one backbone family, which limits the visual information and skills they can capture; combining multiple heterogeneous encoders is non-trivial because their feature structures and spatial/temporal shapes differ.
Approach
Align features from multiple encoders spatio-temporally with parameter-free adaptive 2D average pooling plus per-encoder linear projection to the LLM dimension, then fuse them additively via a single-query cross-attention whose weights depend on visual input; encoders run in parallel under FSDP.
Key results
- MERV achieves 42.28% on TVQA vs 37.66% for Video-LLaVA (+4.62 points) under the same data mix, with MERV (frozen) outperforming Video-LLaVA on most benchmarks.
- MERV (full) reaches 48.4% on Perception Test versus 46.2% for SeViLA, a zero-shot result.
- Single-encoder variants (LanguageBind, DINOv2, ViViT, SigLIP) score 46.81-52.75 average accuracy vs 55.64 for the four-encoder MERV; ViViT alone gets 39.77% on the SSv2 temporal MCQ subset, 9.19 points above LanguageBind.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multiple specialized visual encoders with aligned features and cross-attention fusion instead of the Video-LLaVA single-encoder setup | MERV (frozen) · TVQA | accuracy: 37.66 -> 42.28 | Helped |
| Use multiple specialized visual encoders with aligned features and cross-attention fusion instead of the Video-LLaVA single-encoder setup | MERV (frozen) · VLEP | accuracy: 61.21 -> 58.66 | Hurt |
| Use Stage 1 captioning pre-training and unfreeze the LLM during Stage 1 (ablation) | MERV (frozen) · Perception Test | accuracy: 46.21 -> 48.41 | Helped |
| Mix Stage 1 and Stage 2 training data in a single stage rather than explicit two-stage training (ablation) | MERV · ActivityNet-QA | accuracy: 49.93 -> 39.98 | Hurt |
| Use 16 frames rather than 8 with 2D average pooling (ablation) | MERV pre-fusion projector · MSVD-QA | accuracy: 68.23 -> 69.08 | Helped |
| Use a 2D average pooling projector rather than a 2D attention resampler (ablation) | MERV pre-fusion projector · MSVD-QA | accuracy: 65.76 -> 69.08 | Helped |
| Use 64 rather than 1 pre-fusion output token per frame (ablation) | MERV pre-fusion projector · MSVD-QA | accuracy: 61.94 -> 69.08 | Helped |
| Use cross-attention rather than sequence-wise concatenation for feature fusion (ablation) | MERV · MSVD-QA | accuracy: 66.99 -> 70.97 | Helped |
| Use cross-attention rather than channel-wise concatenation for feature fusion (ablation) | MERV · MSVD-QA | accuracy: 70.02 -> 70.97 | Helped |
| Add LanguageBind to the encoder set (ablation) | MERV with LanguageBind removed · MSVD-QA | accuracy: 68.52 -> 70.97 | Helped |
| Add DINOv2 to the encoder set (ablation) | MERV with DINOv2 removed · MSVD-QA | accuracy: 69.75 -> 70.97 | Helped |
| Add ViViT to the encoder set (ablation) | MERV with ViViT removed · MSVD-QA | accuracy: 70.12 -> 70.97 | Helped |
Surprising findings
- Parameter-free 2D average pooling of features outperformed using full 257-token features without a projector (55.86% vs 54.76% average accuracy).
- Adding a Stage 1 pre-training phase gave little benefit when training only projectors and fusion; mixing Stage 1 and Stage 2 data in one stage performed worse than the two-stage recipe.
Limitations
- Performance is heavily dependent on the capabilities of the underlying LLM (LLaMA-2 7B), and larger models demand more compute.
- If individual encoders are weak at video understanding, MERV cannot fully compensate.
- Limited scale of experiments due to training-time constraints restricts the number of design ablations.
Experimental setup
Datasets: LAION (LLaVA-filtered) Valley LLaVA instruction data Video-ChatGPT instructions MSVD-QA MSRVTT-QA TGIF-QA ActivityNet-QA NExT-QA VLEP TVQA Perception Test
Benchmarks: MSVD-QA MSRVTT-QA TGIF-QA ActivityNet-QA NExT-QA VLEP TVQA Perception Test Something-Something v2 MCQ
Metrics: accuracy score
Baselines: Video-LLaVA LLaMA-VID-7B LLaMA-VID-13B SeViLA Video-ChatGPT Video-Chat LLaMA-Adapter Video-LLaMA
Builds on: Video-LLaVA LLaVA Prismatic VLMs LLaMA-2 FSDP
Closest papers by meaning
- Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.89)
- Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs (NeurIPS 2026 · similarity 0.89)
- A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Frames (CVPR 2024 · similarity 0.87)
- Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders (ICLR 2026 · similarity 0.87)
- MoVA: Adapting Mixture of Vision Experts to Multimodal Context (NeurIPS 2024 · similarity 0.87)
- DejaVid: Encoder-Agnostic Learned Temporal Matching for Video Classification (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.