Research map / Vision-language and multimodal / Multimodal LLMs

Prompt Highlighter: Interactive Control for Multi-Modal LLMs

CVPR 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 11 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Prompt Highlighter, a training-free inference method that lets users highlight prompt tokens or image regions and steers multi-modal LLM generation via classifier-free-style guidance on token probabilities plus attention activation on highlighted tokens.

Problem

Multi-modal LLMs are heavily conditioned on prompt contents, and prompt engineering is unreliable and non-expert-friendly; there is no fine-grained, token-level way for users to control which parts of the input the generation focuses on.

Approach

Form normal and unconditional context pairs by scaling embeddings of highlighted tokens (classifier-free guidance applied to next-token logits), and additionally reweight attention scores toward highlighted tokens (attention activation), including inside Q-Former cross-attention for query-based VLMs.

Key results

• On LLaVA-v1.5 13B without training, Prompt Highlighter reaches 1552.5 MME perception (vs 1531.3) and 69.5 MMBench test (vs 67.0), with 70.7 on MMBench test in the submitted setting.
• On MSCOCO caption evaluation, S-CLIP improves from 0.809 (LLaVA-v1.5) to 0.829, and image-to-text R@1 rises from 50.7 to 62.2.
• A user study over 255 valid preference results found 77.3% of users preferred Prompt Highlighter outputs over original inference baselines.

Measured effects reported in the paper

Surprising findings

• Attention-only ablation gave the largest single gain on MME and MMBench-dev, while guidance alone gave only a small improvement (1528.7 to 1531.1 on MME).
• Highlighting the entire image can mitigate hallucination, lifting POPE from 85.9 to 87.8 on LLaVA-v1.5.

Limitations

• Requires an extra unconditional decoding branch, adding computational and memory overhead (about 7% memory and 10% speed cost in batched caption setting per the appendix).
• Output quality depends on the base model; poorly trained base models may over- or under-emphasize highlighted content, and it cannot add new capabilities.
• Hyperparameters (alpha, beta, gamma) trade off concentration against general VQA performance and were tuned per task.

Experimental setup

Datasets: MSCOCO MME MMBench POPE

Benchmarks: MME MMBench POPE MSCOCO Karpathy test ScienceQA

Metrics: MME perception score MMBench accuracy POPE accuracy S-CLIP CLIP text-image retrieval R@1/R@5 user preference rate tokens/s GPU memory

Baselines: LLaVA-v1.5 InstructBLIP Vicuna-13B LLM-CFG CoCa-CFG FuseCap QWen-VL-Chat mPLUG-Owl-2 MMICL Sphinx InternLM-VLComposer

Builds on: classifier-free guidance LLM-CFG LLaVA BLIP-2

Closest papers by meaning

• Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception (CVPR 2024 · similarity 0.88)
• Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.88)
• ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models (NeurIPS 2024 · similarity 0.88)
• TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.88)
• LLMs can see and hear without any training (ICML 2025 · similarity 0.88)
• MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (ICLR 2024 · similarity 0.88)