Research map / Vision-language and multimodal / Multimodal LLMs
Leveraging Image as Compressed Visual Prompt and Hierarchical Visual Knowledge for Effective Image Utilization in MLLMs
Shezheng Song, Kangcheng Ding, Shan Zhao, Shasha Li, Xiaopeng Li, Chengyu Wang, Qian Wan, Bin Ji, Jie Yu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a method for using images in MLLMs as compressed visual prompts combined with hierarchical visual knowledge, to improve efficient image utilization (full text unavailable; details from abstract only).
Problem
Efficiently utilizing images in multimodal LLMs is hard because full image features create excessive visual tokens that disrupt text, and storing image features as visual knowledge is also suboptimal.
Approach
Treats the image as a compressed visual prompt and pairs it with hierarchical visual knowledge, rather than feeding all image features as prompt tokens or storing them as flat visual knowledge.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| compressed visual prompt plus hierarchical visual knowledge (IPK) vs MemVP | LLaMA-7B MLLM · QA average (5 datasets) | average accuracy (%): 68.62 -> 71.32 | Improved |
| IPK vs MemVP on dialogue and reasoning | LLaMA-7B MLLM · MMBench / POPE | accuracy (%): MMBench 83.3 -> 85.4 | Improved |
| IPK vs Qwen2-VL on MME-Person | LLaMA-7B MLLM · MME-Person | score: 1939.0 vs 1784.6 | Hurt |
| use abstract (pooled) feature instead of complete features as visual prompt (ablation) | IPK input stage · GQA | accuracy (%): 59.75 -> 60.90 | Improved |
| use attention-compressed feature instead of abstract feature as visual prompt (ablation) | IPK input stage · GQA | accuracy (%): 60.9 -> 66.2 | Improved |
| remove long complete visual prompt from input (knowledge only) (ablation) | IPK variant · GQA | accuracy (%): 59.75 -> 60.50 | Improved |
| add hierarchical visual knowledge to FFN (ablation) | IPK variant without prompt · GQA | accuracy (%): 58.10 -> 60.50 | Improved |
| compress the FFN visual knowledge (abstract or compressed) instead of complete knowledge (ablation) | IPK · GQA/VQAv2/VizWiz | accuracy (%): GQA 66.2 -> 62.6 / 61.1 | Hurt |
| increase wavelet level from 0 to 2 (ablation) | IPK · GQA and VizWiz | accuracy and training time | Improved |
| IPK training speed vs MemVP | IPK · average training time | s/it: 0.30 -> 0.32 | No clear effect |
Closest papers by meaning
- LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.91)
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.90)
- Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference (CVPR 2025 · similarity 0.90)
- Hugging Visual Prompt and Segmentation Tokens: Consistency Learning for Fine-Grained Visual Understanding in MLLMs (CVPR 2026 · similarity 0.89)
- Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want (ICLR 2025 · similarity 0.89)
- MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
