Research map / Vision-language and multimodal / Multimodal LLMs
LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LION, an MLLM that adds fine-grained spatial-aware knowledge via a vision aggregator and stage-wise instruction tuning with mixture-of-adapters, plus high-level semantic knowledge via RAM image tags with a soft prompt.
Problem
Existing MLLMs rely on vision encoders pretrained on coarsely aligned image-text pairs, giving insufficient fine-grained and semantic visual knowledge, and naively mixing region-level grounding with image-level VL tasks hurts general performance.
Approach
Injects region-level grounding knowledge through a vision aggregator fusing multi-level ViT features, trained with a three-stage instruction-tuning schedule and a mixture-of-adapters router to resolve conflicts between image-level and region-level tasks; adds RAM image tags via a learnable soft prompt token to mitigate noisy tags.
Key results
- LION-12B reaches 73.77 accuracy on VSR and 108.76 CIDEr on TextCaps, versus 69.93 and 104.17 for re-implemented InstructBLIP (T5XL) with the same data.
- On RefCOCOg zero-shot, LION-12B achieves 66.12 (val) and 64.69 (test) vs Kosmos-2's 60.57 and 61.65.
- Fine-tuned REC average across 8 sets is 86.36 for LION-12B, above Ferret-13B (85.57) and MiniGPTv2 (84.29).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use stage-wise rather than single-stage instruction tuning (ablation) | LION-4B · REC Avg. | accuracy: 3.78 -> 54.46 | Helped |
| Use stage-wise rather than single-stage instruction tuning (ablation) | LION-4B · Image-Level Held-in | score: 84.79 -> 88.07 | Helped |
| Add router to stage-wise instruction tuning (ablation) | LION-4B · Image-Level Held-out | score: 61.91 -> 62.16 | Helped |
| Add router to stage-wise instruction tuning (ablation) | LION-4B · Image-Level Held-in | score: 88.07 -> 87.67 | Hurt |
| Add visual grounding tasks (ablation) | LION-4B · OKVQA | accuracy: 47.38 -> 51.07 | Helped |
| Add visual grounding tasks (ablation) | LION-4B · GQA | accuracy: 46.34 -> 49.22 | Helped |
| Add image tags with soft prompting (ablation) | LION-4B with visual grounding · AOKVQA | accuracy: 56.90 -> 59.98 | Helped |
| Add image tags with soft prompting (ablation) | LION-4B with visual grounding · Flickr30K | CIDEr: 83.99 -> 85.57 | Helped |
| Add image tags with soft prompting (ablation) | LION-4B with visual grounding · VSR | accuracy: 73.20 -> 72.96 | Hurt |
| Add visual grounding tasks and image tags (ablation) | LION-4B · REC Avg. | accuracy | Helped |
| Use the vision aggregator (ablation) | LION-4B · RefCOCO, RefCOCO+, and RefCOCOg | accuracy | Helped |
| Use stage-wise instruction tuning rather than single-stage training (ablation) | LION-4B · Image-Level Held-out | score: 60.20 -> 61.91 | Helped |
Surprising findings
- Single-stage instruction tuning with region-level data collapses REC average to 3.78, while stage-wise training recovers it to 54.46.
Limitations
- Depends on an off-the-shelf tagger (RAM) whose errors can propagate; soft prompt only partially mitigates them.
- Region-level training relies on Visual Genome; zero-shot gains are smaller on RefCOCO/RefCOCO+ test-A sets.
- Stage-wise schedule adds training complexity and the router slightly reduces held-in performance.
Experimental setup
Datasets: COCO Caption TextCaps OKVQA A-OKVQA VQAv2 OCR-VQA LLaVA-Instruct-150K Visual Genome RefCOCO RefCOCO+ RefCOCOg Flickr30K
Benchmarks: Flickr30K COCOCap TextCaps OKVQA AOKVQA GQA IconQA VSR VisDial RefCOCO RefCOCO+ RefCOCOg
Metrics: CIDEr top-1 accuracy MRR F1-score accuracy
Baselines: InstructBLIP Kosmos-2 Shikra Pink Ferret MiniGPT-4 LLaVA MiniGPTv2 POPE baselines (mPLUG-Owl)
Builds on: InstructBLIP BLIP-2 Q-Former Shikra instruction templates recognize anything model (RAM) Mixture-of-Experts
Closest papers by meaning
- UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm (CVPR 2026 · similarity 0.92)
- Hugging Visual Prompt and Segmentation Tokens: Consistency Learning for Fine-Grained Visual Understanding in MLLMs (CVPR 2026 · similarity 0.92)
- Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception (CVPR 2024 · similarity 0.91)
- Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs (CVPR 2024 · similarity 0.91)
- Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.91)
- ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.