Research map / Multi-agent systems and games / LLM agents
MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Multimodal Role-Playing Agents with the MMRole framework: a 85-character image-grounded role-play dataset, a reward-model-based evaluation (MMRole-Eval) over eight metrics, and MMRole-Agent, a fine-tuned Qwen-VL-Chat.
Problem
Role-playing agents are limited to text, while humans perceive through vision too; existing RPA evaluation scores outputs without ground truth, which is unstable.
Approach
Defines multimodal role-playing (dialogue centered on images conditioned on character profiles), builds a GPT-4-synthesized image-grounded dataset, and scores agents relative to ground-truth responses using a trained pairwise score-ratio reward model instead of direct scoring.
Key results
- MMRole-Agent reaches overall score 0.994 on MMRole-Eval, up 0.151 over its base Qwen-VL-Chat (0.843), and is best among 9B-scale models.
- Claude 3 Opus achieves the top overall score of 1.157 among MRPAs over 100B; LLaVA-NeXT-34B (1.027) leads the tens-of-billions group.
- Reward model scoring success rate is 100% versus 33.13% for base Qwen-VL-Chat, with overall MAE 0.0738 against GPT-4 scores; human-gap MAE of reward model (0.1258) is comparable to GPT-4 (0.1254).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune QWen-VL-Chat on MMRole-Data | QWen-VL-Chat · MMRole test | Overall: 0.843 -> 0.994; +0.151 | Helped |
| Exclude image information from the input (ablation) | MMRole-Agent · Overall | Image-Text Relevance: 0.9875 -> 0.7003 | Hurt |
| Exclude image information from the input (ablation) | GPT-4 Turbo · Overall | Image-Text Relevance: 1.1995 -> 0.9306 | Hurt |
| Exclude image information from the input (ablation) | Claude 3 Opus · Overall | Image-Text Relevance: 1.2298 -> 0.8838 | Hurt |
| Train on all 85K samples instead of 8.5K (ablation) | MMRole-Agent · MMRole test | Overall: 0.967 -> 0.989 | Helped |
| Add other English fictional characters to The Avengers training set (ablation) | MMRole-Agent · MMRole Out-Test | Overall: 0.965 -> 0.968 | Helped |
| Add hypothetical real-life characters to The Avengers training set (ablation) | MMRole-Agent · MMRole Out-Test | Overall: 0.965 -> 0.970 | Helped |
| Train with all 72 characters rather than The Avengers only (ablation) | MMRole-Agent · MMRole Out-Test | Overall: 0.965 -> 0.983 | Helped |
| Modify the system prompt wording (ablation) | MMRole-Agent · MMRole test | Overall: 0.994 -> 0.995 | No clear effect |
| Modify the character-designating prompt wording (ablation) | MMRole-Agent · MMRole test | Overall: 0.994 -> 0.996 | No clear effect |
| Use the dedicated role-playing assistant prompt rather than the simpler identity prompt (ablation) | GPT-4 dialogue generation · MMRole-Data dialogue generation | Dialogue generation quality | Helped |
| Use MMRole-Agent rather than QWen-VL-Chat | QWen-VL-Chat · MMRole test | Overall: 0.843 -> 0.994 | Helped |
Surprising findings
- Removing images drops Image-Text Relevance sharply for all RPAs, most strongly in single-turn commentary scenarios.
- Smaller open models such as LLaVA-NeXT-Mistral-7B and MMRole-Agent surpass Yi-VL-34B, suggesting training data and methods matter more than scale.
Limitations
- Training data is synthesized primarily by GPT-4, which may cap performance relative to GPT-4 itself.
- Cronbach's alpha of 0.70 indicates only moderate internal consistency of MMRole-Eval.
- Evaluation relies on GPT-4-generated ground truth and GPT-4-derived reward model labels, and human validation covers only a small 20-question set.
Experimental setup
Datasets: MMRole-Data MS-COCO
Benchmarks: MMRole-Eval
Metrics: Instruction Adherence Fluency Coherency Image-Text Relevance Response Accuracy Personality Consistency Knowledge Consistency Tone Consistency MAE RMSE
Baselines: GPT-4 Turbo Gemini Pro Vision Claude 3 Opus QWen-VL-Max LLaVA-NeXT-34B Yi-VL-34B InternVL-Chat-V1.5 QWen-VL-Chat LLaVA-NeXT-Mistral-7B Yi-VL-6B
Builds on: Qwen-VL LLaVA Vicuna MT-Bench evaluation
Closest papers by meaning
- DMT-RoleBench: A Dynamic Multi-Turn Dialogue Based Benchmark for Role-Playing Evaluation of Large Language Model and Agent (AAAI 2025 · similarity 0.87)
- RoleAgent: Building, Interacting, and Benchmarking High-quality Role-Playing Agents from Scripts (NeurIPS 2024 · similarity 0.87)
- Thinking in Character: Advancing Role-Playing Agents with Role-Aware Reasoning (NeurIPS 2025 · similarity 0.86)
- CoSER: Coordinating LLM-Based Persona Simulation of Established Roles (ICML 2025 · similarity 0.85)
- MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models (ICLR 2026 · similarity 0.85)
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.