Research map / Vision-language and multimodal / Human and embodied modeling
CogVLM: Visual Expert for Pretrained Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CogVLM, an open-source VLM that adds a trainable visual expert (separate QKV and FFN per layer) to a frozen-initialized Vicuna-7B for deep vision-language fusion, reaching SOTA on many captioning, VQA, LVLM and grounding benchmarks.
Problem
Shallow alignment VLMs that map image features into a frozen LLM's input space underperform, while fully fine-tuning the LLM for VLM training causes catastrophic forgetting of NLP ability.
Approach
A trainable visual expert module with its own QKV matrices and FFN in every layer processes image tokens, initialized from the LLM, so image and text get deep fusion while the original language weights stay frozen and text-only behavior is preserved.
Key results
- CogVLM-Chat (Vicuna-7B) reports MM-Vet 51.1, SEED 72.5, MMBench 77.6, POPE 87.9, MMMU 41.1 and MathVista 34.5, the best among the listed generalist models.
- CogVLM-Grounding reaches 92.76 on RefCOCO val and 90.79 on RefCOCOg test, with SOTA on 5 of 9 grounding splits.
- On Flickr30K captioning CogVLM scores 94.9 CIDEr, and on NoCaps val overall 128.3, using 1.5B pretraining pairs versus 12.9B for GIT2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use deep visual expert fusion instead of shallow adapter-only alignment (ablation) | CogVLM · COCO Karpathy test | CIDEr: 131.2 -> 142.8 | Helped |
| Use visual expert fusion instead of training LLM parameters with the adapter (ablation) | CogVLM · COCO Karpathy test | CIDEr: 140.3 -> 142.8 | Helped |
| Use visual expert in every layer rather than every 4th layer (ablation) | CogVLM · COCO Karpathy test | CIDEr: 138.7 -> 142.8 | Helped |
| Add visual expert attention alongside the FFN expert (ablation) | CogVLM · COCO Karpathy test | CIDEr: 140.0 -> 142.8 | Helped |
| Initialize the visual expert from LLM weights rather than random initialization (ablation) | CogVLM · COCO Karpathy test | CIDEr: 138.0 -> 142.8 | Helped |
| Use a causal visual attention mask rather than a full mask (ablation) | CogVLM · COCO Karpathy test | CIDEr: 141.0 -> 142.8 | Helped |
| Add image self-supervised loss (ablation) | CogVLM · TextVQA | top1: 45.3 -> 45.9 | No clear effect |
| Use EVA2-E rather than EVA2-L as visual encoder (ablation) | CogVLM · TextVQA | top1: 42.8 -> 45.3 | Helped |
| Use EMA during pretraining (ablation) | CogVLM · COCO Karpathy test | CIDEr: 143.1 -> 142.8 | No clear effect |
Surprising findings
- Causal mask on visual tokens works better than a full bidirectional mask, despite the latter giving more information.
- Adding a self-supervised image CLIP-feature loss gave no downstream improvement at 6.6B scale, and a smaller EVA2-L encoder was only slightly worse on most tasks.
Limitations
- Visual expert roughly doubles parameters (6.6B trainable in ablations), adding memory cost.
- Shallow-alignment and other variants were only compared in ablations at limited scale; the paper notes VLM training is still early, with no RLHF or anti-hallucination work.
Experimental setup
Datasets: LAION-2B COYO-700M LAION-115M VQAv2 OKVQA TextVQA OCRVQA ScienceQA LLaVA-Instruct LRV-Instruction LLaVAR Flickr30K Entities
Benchmarks: NoCaps Flickr30K COCO TextCaps VQAv2 OKVQA TextVQA OCRVQA ScienceQA MM-Vet SEED-Bench MMBench
Metrics: CIDEr VQA score accuracy GPT-4 score top-1 accuracy
Baselines: InstructBLIP MiniGPT-4 LLaVA-1.5 Qwen-VL GIT2 PaLI-X-55B PaLI-17B Emu2-Chat mPLUG-Owl2 SPHINX-2k Shikra-13B Ferret-13B
Builds on: BLIP-2 LLaVA-1.5 LoRA P-Tuning PaLI-X
Closest papers by meaning
- Probing Visual Language Priors in VLMs (ICML 2025 · similarity 0.89)
- VILA: On Pre-training for Visual Language Models (CVPR 2024 · similarity 0.88)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.88)
- SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding (CVPR 2025 · similarity 0.88)
- CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding (ICLR 2024 · similarity 0.88)
- Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.