Research map / Vision-language and multimodal / Cross-modal learning
Libra: Building Decoupled Vision System on Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Libra builds a decoupled vision system on a frozen LLaMA2-7B-Chat via a routed visual expert with a cross-modal bridge, trained with discrete auto-regressive image modeling on 50M image-text pairs.
Problem
Existing LLM-based MLLMs rely on a pretrained vision encoder with limited visual understanding, while unified or contiguous image-modeling approaches (e.g., Emu) lose unique visual information or suffer from unstable learning; the paper seeks a better vision system on top of LLMs for image-to-text c
Approach
A routed visual expert gives vision tokens separate QKV and FFN parameters alongside the frozen LLM, and a low-rank cross-modal bridge adds learnable projections to keys and values only for cross-modal attention, decoupling inner-modal modeling from cross-modal interaction; vision is trained with discrete LFQ-based tokens from a CLIP-based tokenizer, hybridized with contiguous CLIP features.
Key results
- Libra (11.3B, 50M pretraining pairs) achieves 59.5 on VizWiz versus 38.9 for Qwen-VL, and 88.2 on POPE, the best among compared MLLMs.
- Libra reaches 30.0 on MMVP, +5.3% over LLaVA1.5, and 62.7 on SEED.
- Removing discrete image modeling (language-only supervision) or switching to contiguous modeling reduces performance; contiguous supervision gives -6.2% on VizWiz relative to Libra.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace unified image-and-language modeling with language-only modeling (ablation) | Libra · VizWiz | VQA Score: 52.9 -> 59.5 | Helped |
| replace discrete image supervision with contiguous image supervision (ablation) | Libra · VizWiz | VQA Score: 53.3 -> 59.5 | Helped |
| increase pretraining data from 3M to 50M (ablation) | Libra · VizWiz | VQA Score: 48.8 -> 59.5 | Helped |
| add the routed visual expert (ablation) | Libra · VizWiz | VQA Score: 50.8 -> 59.5 | Helped |
| add the cross-modal bridge (ablation) | Libra with routed visual expert · VizWiz | VQA Score: 53.6 -> 59.5 | Helped |
| add contiguous visual signals to discrete inputs (ablation) | Libra · VizWiz | VQA Score: 38.6 -> 59.5 | Helped |
| use the CLIP vision encoder instead of training the vision encoder from scratch (ablation) | Libra · VizWiz | VQA Score: 40.6 -> 59.5 | Helped |
| use Libra instead of Qwen-VL | Qwen-VL · VizWiz | VQA Score: +20.6% | Helped |
| use Libra instead of LLaVA1.5 | LLaVA1.5 · MMVP | GPT-4 Score: +5.3% | Helped |
Surprising findings
- A simple visual expert without the cross-modal bridge fails at image modeling, producing an invalid vision system.
- Contiguous image modeling performs similarly to no image modeling at all, contrary to the benefits expected from visual self-supervision.
Limitations
- Inherits hallucinations and poor length generalization from the underlying LLM.
- The routed visual expert attention mechanism is not supported by existing acceleration frameworks such as FlashAttention.
- Not state-of-the-art; fails at complex text-to-image generation given limited training tokens (7B).
Experimental setup
Datasets: COYO-700M CC12M COCO LLaVA-Instruct VQAv2 OKVQA GQA VizWiz SQA NoCaps Flickr30k POPE
Benchmarks: VQAv2 OKVQA GQA VizWiz SQA NoCaps Flickr30k COCO POPE MME MMBench MMBench-CN
Metrics: VQA score EM accuracy accuracy CIDEr F1 MME score GPT-4 score
Baselines: BEiT-3 PaLI-X OFA CogVLM BLIP-2 Flamingo Unified-IO XL PaLM-E InstructBLIP Emu Qwen-VL Shikra
Builds on: CogVLM LLaVA VQGAN lookup-free quantization LLaMA2
Closest papers by meaning
- InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (CVPR 2024 · similarity 0.87)
- Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs (CVPR 2024 · similarity 0.87)
- L-Man: A Large Multi-modal Model Unifying Human-centric Tasks (AAAI 2025 · similarity 0.87)
- Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.87)
- A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models (ICLR 2026 · similarity 0.87)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.