AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

Leveraging Image as Compressed Visual Prompt and Hierarchical Visual Knowledge for Effective Image Utilization in MLLMs

Shezheng Song, Kangcheng Ding, Shan Zhao, Shasha Li, Xiaopeng Li, Chengyu Wang, Qian Wan, Bin Ji, Jie Yu

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a method for using images in MLLMs as compressed visual prompts combined with hierarchical visual knowledge, to improve efficient image utilization (full text unavailable; details from abstract only).

Problem

Efficiently utilizing images in multimodal LLMs is hard because full image features create excessive visual tokens that disrupt text, and storing image features as visual knowledge is also suboptimal.

Approach

Treats the image as a compressed visual prompt and pairs it with hierarchical visual knowledge, rather than feeding all image features as prompt tokens or storing them as flat visual knowledge.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
compressed visual prompt plus hierarchical visual knowledge (IPK) vs MemVPLLaMA-7B MLLM · QA average (5 datasets)average accuracy (%): 68.62 -> 71.32Improved
IPK vs MemVP on dialogue and reasoningLLaMA-7B MLLM · MMBench / POPEaccuracy (%): MMBench 83.3 -> 85.4Improved
IPK vs Qwen2-VL on MME-PersonLLaMA-7B MLLM · MME-Personscore: 1939.0 vs 1784.6Hurt
use abstract (pooled) feature instead of complete features as visual prompt (ablation)IPK input stage · GQAaccuracy (%): 59.75 -> 60.90Improved
use attention-compressed feature instead of abstract feature as visual prompt (ablation)IPK input stage · GQAaccuracy (%): 60.9 -> 66.2Improved
remove long complete visual prompt from input (knowledge only) (ablation)IPK variant · GQAaccuracy (%): 59.75 -> 60.50Improved
add hierarchical visual knowledge to FFN (ablation)IPK variant without prompt · GQAaccuracy (%): 58.10 -> 60.50Improved
compress the FFN visual knowledge (abstract or compressed) instead of complete knowledge (ablation)IPK · GQA/VQAv2/VizWizaccuracy (%): GQA 66.2 -> 62.6 / 61.1Hurt
increase wavelet level from 0 to 2 (ablation)IPK · GQA and VizWizaccuracy and training timeImproved
IPK training speed vs MemVPIPK · average training times/it: 0.30 -> 0.32No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy