Research map / Vision-language and multimodal / Multimodal LLMs

Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding

CVPR 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 111 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Chat-UniVi unifies image and video understanding in an LLM by merging CLIP visual tokens into multi-scale dynamic tokens via parameter-free DPC-KNN clustering, trained jointly on mixed image and video instruction data.

Problem

Existing multimodal LLMs specialize in either images (many visual tokens) or videos (many frames with few tokens per frame), and struggle to handle both within a unified, token-efficient framework.

Approach

Represent images and videos with a shared set of dynamic visual tokens produced by parameter-free density-peaks (DPC-KNN) clustering that merges spatially similar tokens and temporally clusters frames into events, plus multi-scale features concatenated across three merging steps.

Key results

• On image conversation GPT evaluation, Chat-UniVi (7B, 112 visual tokens) reaches an All score of 84.2 vs 80.4 for the re-implemented LLaVA-7B (256 tokens) and 85.1 for LLaVA-13B.
• On zero-shot video QA, Chat-UniVi gets 55.0 on MSRVTT-QA, 69.3 on MSVD-QA, 69.0 on TGIF-QA and 46.1 on ActivityNet-QA, above Video-ChatGPT (49.3, 64.9, 51.4, 35.2).
• On POPE random split, multi-scale Chat-UniVi reaches 85.19 accuracy versus 72.16 for re-implemented LLaVA-7B and 79.67 for MiniGPT-4 (13B).

Measured effects reported in the paper

Surprising findings

• Training on only images or only videos, or sequential image-then-video training, degrades the other modality, while joint mixed training improves both.
• The parameter-free clustering outperformed a parameterized clustering variant, which the authors attribute to gradient instability in multimodal conversation training.

Limitations

• Performance declines when multiple videos are input, possibly due to sequence length limits.
• Multi-turn conversations show forgetfulness of prior context; inherits LLM hallucination and prompt sensitivity issues.
• Natural language output is less suitable for structured outputs such as bounding boxes or dense predictions.

Experimental setup

Datasets: CC3M-595K COCO LLaVA-instruct-150K MIMIC-IT Video-ChatGPT-instruct ScienceQA MSRVTT-QA MSVD-QA TGIF-QA ActivityNet-QA POPE

Benchmarks: GPT-4 image conversation evaluation (COCO 90 questions) ActivityNet-based video GPT evaluation ScienceQA MSRVTT-QA MSVD-QA TGIF-QA ActivityNet-QA POPE

Metrics: GPT score (0-10 / normalized 0-100) accuracy F1-score yes ratio human score (0-10)

Baselines: LLaVA MiniGPT-4 InstructBLIP mPLUG-Owl MultiModal-GPT Video-ChatGPT VideoChat Video-LLaMA LLaMA-Adapter FrozenBiLM LLaMA-SciTune

Builds on: LLaVA DPC-KNN Vicuna-v1.5 Video-ChatGPT MIMIC-IT

Closest papers by meaning

• Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.90)
• PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.89)
• The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models (NeurIPS 2025 · similarity 0.89)
• SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models (NeurIPS 2025 · similarity 0.89)
• What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
• What's in the Image? A Deep-Dive into the Vision of Vision Language Models (CVPR 2025 · similarity 0.89)