Research map / Vision-language and multimodal / Multimodal LLMs

ModularAgent: A Task-Aware Modular Framework for Joint Optimization of Multimodal Large Language Models and World Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ModularAgent, which couples an MLLM (InternVideo2 backbone) and an RSSM world model via task-aware modular fusion and text-conditioned dense rewards, evaluated on DMC locomotion tasks and cross-environment transfer.

Problem

Existing MLLM-world model integrations are one-way and task-agnostic, so semantic intent and latent dynamics are not tightly coupled, and multi-task and cross-environment generalization suffers.

Approach

A task-conditioned gated modular fusion with layer-wise semantic and dynamics adapters injects MLLM features into the world model latent space, and dense cosine-similarity reward plus joint loss backpropagates imagination feedback into the MLLM for bidirectional coupling.

Key results

• ModularAgent achieves the highest score on 9 of 10 DMC tasks, with overall normalized score 0.91 ± 0.02 versus 0.87 for FOUNDER and 0.82 for GenRL.
• On Walker run, ModularAgent improves from 0.78 (GenRL/FOUNDER range) to 0.87.
• Ablation on Walker: base 0.89/0.67/0.86 rises to 1.03/0.87/1.02 across stand/run/walk with the full model (LMLLM, TAMF, joint optimization).

Limitations

• Evaluated only on four locomotion environments from DMC with 64x64 renders.
• FOUNDER official implementation unavailable, so only WM-CLIP and GenRL compared in cross-domain setting.
• Hardware and training compute not reported.

Experimental setup

Datasets: DeepMind Control Suite offline datasets (Plan2Explore and replay buffers)

Benchmarks: DeepMind Control Suite (Walker, Cheetah, Quadruped, Stickman)

Metrics: normalized episodic reward (min-max) cross-environment transfer score

Baselines: TD3 IQL TD3+BC WM-CLIP GenRL FOUNDER

Builds on: GenRL DreamerV3 FOUNDER RSSM

Closest papers by meaning

• EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.92)
• From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons (CVPR 2025 · similarity 0.91)
• GenRL: Multimodal-foundation world models for generalization in embodied agents (NeurIPS 2024 · similarity 0.91)
• Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld (CVPR 2024 · similarity 0.90)
• On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.90)
• World Model Implanting for Test-time Adaptation of Embodied Agents (ICML 2025 · similarity 0.89)