Research map / Vision-language and multimodal / Multimodal LLMs
ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ControlMLLM, a training-free method that injects box, mask, scribble, or point visual prompts into frozen MLLMs by optimizing a learnable latent added to visual tokens at test time with an attention-based energy function.
Problem
Referring (region-level) abilities in MLLMs usually require costly fine-tuning on region-text data, which hurts flexibility and out-of-domain generalization when domains or base models change.
Approach
Attention between a context text token and visual tokens is pooled and an energy function pushes attention mass into the referred region; a zero-initialized latent added to the MLP-projected visual tokens is optimized at the first decoding step at test time, with EMA and early stopping to limit overfitting.
Key results
- On the box Referring Object Classification task, LLaVA + Ours reaches 60.59% vs 54.72% for vanilla LLaVA and 58.59% for GPT4-ROI, and beats Shikra-7B on point referring (58.85 vs 56.27).
- On the out-of-domain Referring Text Classification task, LLaVA + Ours reaches 61.22% box accuracy vs 53.57% vanilla LLaVA and 55.47% for Ferret-7B.
- The method applies to InstructBLIP-7B and LLaVA-HR-7B, with ROC gains from 49.81 to 54.91 and 53.81 to 58.92 respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| optimize visual tokens using a latent variable | LLaVA-v1.5-7B · ROC test set, box | accuracy: 54.72 -> 60.59 | Helped |
| optimize visual tokens using a latent variable | LLaVA-v1.5-7B · RTC, box | accuracy: 53.57 -> 61.22 | Helped |
| optimize visual tokens using a latent variable | LLaVA-1.5-7B · RefCOCOg box referring description | METEOR: 13.15 -> 14.00 | Helped |
| use T=3 optimization steps rather than T=0 (ablation) | LLaVA-v1.5-7B · ROC validation set, box | accuracy: 57.00 -> 61.00 | Helped |
| increase optimization steps from T=3 to T=4 at α=400 (ablation) | LLaVA-v1.5-7B · ROC validation set, box | accuracy: 61.00 -> 53.50 | Hurt |
| use EMA with β=0.3 instead of β=0.5 (ablation) | LLaVA-v1.5-7B · ROC validation set, box, T=4 | accuracy: 60.00 -> 62.00 | Helped |
| use EMA with β=0.7 instead of β=0.5 (ablation) | LLaVA-v1.5-7B · ROC validation set, box, T=4 | accuracy: 60.00 -> 54.00 | Hurt |
| add Early Stop (ablation) | LLaVA-v1.5-7B · ROC validation set, box | accuracy: 57.00 -> 62.50 | Helped |
| increase optimization steps from T=4 to T=5 with Early Stop (ablation) | LLaVA-v1.5-7B · ROC validation set, box | accuracy: 60.50 -> 62.50 | Helped |
| increase optimization steps from T=3 to T=4 at α=500 (ablation) | LLaVA-v1.5-7B · ROC validation set, box | accuracy: 56.00 -> 43.00 | Hurt |
| increase optimization steps from T=3 to T=4 at α=200 (ablation) | LLaVA-v1.5-7B · ROC validation set, box | accuracy: 60.50 -> 58.00 | Hurt |
Surprising findings
- Naively editing attention with a fixed bias (LLaVA + Edit Att) degrades accuracy well below vanilla LLaVA (36.24 on box ROC), motivating the latent-variable optimization over visual tokens.
- Performance on LLaVA-1.5-13B is worse than on 7B, which the authors attribute to more attention maps making optimization harder.
Limitations
- Adds inference overhead from test-time optimization (T=5 iterations per sample).
- Requires white-box model access and depends on the base MLLM's capabilities; supports only a single region prompt.
- Results are sensitive to text prompt clarity and to visual prompt size, and performance degrades on LLaVA-1.5-13B.
Experimental setup
Datasets: LVIS validation COCO-Text RefCOCOg
Benchmarks: Referring Object Classification (LVIS-based) Referring Text Classification (COCO-Text-based) RefCOCOg referring description
Metrics: accuracy BLEU@4 METEOR CIDEr-D SPICE relevancy score
Baselines: Kosmos-2 GPT4-ROI Shikra-7B Ferret-7B LLaVA + Color LLaVA + Blur LLaVA + Edit Att
Builds on: Ferret Layout-Guidance Prompt-to-Prompt Test-time prompt tuning Attention-based relevancy (DIFNet/IBD)
Closest papers by meaning
- Exploring the Transferability of Visual Prompting for Multimodal Large Language Models (CVPR 2024 · similarity 0.92)
- Wings: Learning Multimodal LLMs without Text-only Forgetting (NeurIPS 2024 · similarity 0.91)
- Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.91)
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.90)
- Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception (CVPR 2024 · similarity 0.90)
- LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.