Research map / Vision-language and multimodal / Multimodal LLMs

Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces multimodal prompt optimization, where prompts pair text with generated images, videos, or molecules, and proposes MPO, which jointly updates both modalities and selects candidates via prior-inherited Bayesian UCB.

Problem

Automatic prompt optimization has been limited to text, even though MLLMs can take images, videos, and molecules as prompt inputs; jointly optimizing text and non-textual prompts creates cross-modal consistency issues and makes good candidates sparse to select.

Approach

Joint textual and non-textual prompt updates driven by a single shared textual failure feedback, with generation/edit/mix operators that call modality-specific generators (e.g., text-to-image, text-to-video, text-to-molecule), plus a Beta-posterior Bayesian UCB selection that initializes child prompts from parent posterior means.</303>

Key results

• MPO achieves 65.1 average score versus 60.0 for the best text-only baseline ProTeGi across 10 datasets.
• On CUB, MPO reaches 78.6 accuracy versus 70.0 for ProTeGi and 71.6 for SEE.
• Prior-inherited Bayesian UCB reaches Uniform-level performance with 30% of the evaluation budget and reduces budget by 42% relative to w/o Prior.

Surprising findings

• Text-only prompt optimization methods like ProTeGi plateau after a few iterations on CUB, while MPO keeps improving.

Limitations

• Video editing and mixing are disabled; only generation operator used for video due to complexity.
• Relies on GPT-4o mini as optimizer and external generators, adding cost (117 generator calls per run in table).

Experimental setup

Datasets: PlantVillage CUB-200-2011 SLAKE DrivingVQA RSVQA Drive&Act VANE-Bench TDC Absorption BBBP CYP Inhibition

Benchmarks: PlantVillage CUB-200-2011 SLAKE DrivingVQA RSVQA Drive&Act VANE-Bench TDC Absorption BBBP CYP Inhibition

Metrics: accuracy F1 DSG score

Baselines: Human CoT Few-Shot APE OPRO EvoPrompt PE2 ProTeGi SEE

Builds on: ProTeGi OPRO EvoPrompt UCB Chain-of-Thought

Closest papers by meaning

• IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.90)
• The Devil is in the Prompts: Retrieval-Augmented Prompt Optimization for Text-to-Video Generation (CVPR 2025 · similarity 0.89)
• Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment (CVPR 2025 · similarity 0.89)
• Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs (NeurIPS 2025 · similarity 0.89)
• ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models (NeurIPS 2024 · similarity 0.89)
• PromptCoT: Align Prompt Distribution via Adapted Chain-of-Thought (CVPR 2024 · similarity 0.89)