Research map / Multi-agent systems and games / LLM agents

Multimodal Policy Internalization for Conversational Agents

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent decision making · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces multimodal policy internalization (MPI), a three-stage TriMPI pipeline (visually-masked continual pretraining, CoT SFT, RL with PolicyRollout) that internalizes reasoning-intensive multimodal policies into Qwen2.5-VL weights, with two new datasets, ClevrPolicy and GTAPolicy.

Problem

Long, reasoning-intensive multimodal policies (system prompts with rules, tool-use instructions, visual demos) impose large fixed prompt costs and are followed unreliably in context; prior prompt compression and policy internalization work covers only short templates or text-only safety specs.

Approach

Combine continual pretraining on the policy text with visual tokens masked from the loss, CoT SFT, and then RL where the rollout group is augmented with responses generated with the policy in context, while the policy gradient is applied only to the no-policy path so training and inference stay aligned.

Key results

• TriMPI w/ PoRo-GRPO reaches 84.70% on ClevrPolicy-M (N=6) and 81.06% overall on GTAPolicy, versus 54.50% for CoT SFT and 21.51% in-context.
• Removing the policy from the prompt cuts prompt tokens by up to 93.9% and prefill time by up to 85.7%.
• Under policy override, TriMPI w/ PoRo-GRPO scores 82.70% on ClevrPolicy-M vs 25.20% for CoT SFT; it also retains 34.76 average on MMMU-Pro/MMLU-Pro vs 26.96 for CoT SFT.

Surprising findings

• Simply inserting the policy in-context during training but removing it at inference yields near-random performance.
• GRPO barely reduces the discrepancy in condition-ID distribution relative to SFT (38.72 vs 39.74 average absolute difference).

Limitations

• Datasets are small, especially GTAPolicy (451 train instances), and real-world image diversity is limited.
• Continual pretraining is only a simple visual-token masking strategy.
• Does not handle mixtures of tasks with very different response formats.

Experimental setup

Datasets: CLEVR GTA

Benchmarks: ClevrPolicy-T ClevrPolicy-M GTAPolicy MMMU-Pro MMLU-Pro WildGuardTest

Metrics: accuracy tool accuracy argument score overall score 0-10 policy referral score prefill time prompt tokens

Baselines: In-Context Direct SFT CoT SFT CoT SFT + GRPO CoT SFT + DAPO Claude-3.7-Sonnet Claude-4-Sonnet

Builds on: GRPO DAPO PromptIntern deliberative alignment CLEVR GTA

Closest papers by meaning

• Simulation-Free Hierarchical Latent Policy Planning for Proactive Dialogues (AAAI 2025 · similarity 0.89)
• Diving into Self-Evolving Training for Multimodal Reasoning (ICML 2025 · similarity 0.87)
• DeepEyesV2: Toward Agentic Multimodal Model (ICLR 2026 · similarity 0.87)
• WorldMemArena: Evaluating Multimodal Agent Memory Through Action–World Interaction (NeurIPS 2026 · similarity 0.87)
• Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents (ICLR 2024 · similarity 0.87)
• On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.86)