Research map / Vision-language and multimodal / Multimodal LLMs
InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces InterMT, a human-preference dataset for multi-turn interleaved vision-language understanding and generation, plus InterMT-Bench for MLLM-as-judge evaluation and a judge model trained on it.
Problem
Existing alignment and preference data focus on single-turn, unimodal or understanding-only settings, leaving multi-turn interleaved multimodal understanding and generation without human preference data and judge benchmarks.
Approach
A tree-structured, tool-augmented agent workflow (MLLMs plus image generation/editing/retrieval tools) builds multi-turn interleaved QA, which is then annotated by humans at turn-level (local) and conversation-level (global) across nine helpfulness sub-dimensions with language feedback.
Key results
- Best MLLM judges reach about 0.24 Pearson on local/global scoring evaluation (GPT-4o avg 0.242 local, 0.153 global), well below human agreement.
- Pair comparison accuracies for top judges hover around 0.53-0.54, showing limited alignment with human preferences.
- Models achieve near-human 4.38/5 on Crucial Step Recognition, while AI-human agreement is about 60% on local evaluations and markedly lower on global ones.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Model fine-grained local (turn-level) preferences rather than directly modeling global conversation-level preferences | InterMT preference modeling · Multi-turn multimodal preference modeling | alignment with human values | Helped |
| Transfer global preferences to local preferences rather than local preferences to global preferences | InterMT preference transfer · Multi-turn multimodal interactions | alignment with actual preferences | Helped |
| Train turn-based judges with more preceding conversation turns | Turn-based judge moderation · Future turns in multi-turn interactions | accuracy | Helped |
| Increase training-turn count for longer evaluation horizons | Turn-based judge moderation · Future turns in multi-turn interactions | generalization ability | Hurt |
| Add reasoning to Gemini-2.0-Flash (ablation) | Gemini-2.0-Flash · InterMT-Bench Scoring Evaluation, local Avg. | Pearson correlation: 0.186 -> 0.181 | Hurt |
| Add reasoning to GPT-4.1 (ablation) | GPT-4.1 · InterMT-Bench Scoring Evaluation, local Avg. | Pearson correlation: 0.218 -> 0.229 | Helped |
| Add reasoning to GPT-4o (ablation) | GPT-4o · InterMT-Bench Scoring Evaluation, local Avg. | Pearson correlation: 0.242 -> 0.228 | Hurt |
| Add reasoning to Gemini-2.5-Pro-preview (ablation) | Gemini-2.5-Pro-preview · InterMT-Bench Scoring Evaluation, local Avg. | Pearson correlation: 0.194 -> 0.215 | Helped |
| Add reasoning to Claude-3.7-Sonnet (thinking) (ablation) | Claude-3.7-Sonnet (thinking) · InterMT-Bench Scoring Evaluation, local Avg. | Pearson correlation: 0.209 -> 0.196 | Hurt |
| Add reasoning to o4-mini (ablation) | o4-mini · InterMT-Bench Scoring Evaluation, local Avg. | Pearson correlation: 0.209 -> 0.214 | Helped |
| Add reasoning to GPT-4.1 (ablation) | GPT-4.1 · InterMT-Bench Scoring Evaluation, global Avg. | Pearson correlation: 0.122 -> 0.191 | Helped |
| Add reasoning to GPT-4o (ablation) | GPT-4o · InterMT-Bench Pair Comparison, local Avg. | accuracy: 0.503 -> 0.514 | Helped |
Surprising findings
- Adding explicit reasoning (+reason) or stronger reasoning models like o4-mini does not reliably improve alignment with human judgments.
- Global-to-local preference transfer is consistently easier than local-to-global transfer.
Limitations
- Only vision-language modalities; excludes video, audio and other modalities.
- AI feedback agreement with humans is low on global evaluations, so AI annotation cannot replace humans yet.
- Judge models show position and high-score biases and hallucination.
Experimental setup
Datasets: LLaVA-Instruct-150K ART500K MovieNet RLHF-V ShareGPT4V MM-RLHF Align-Anything-200K CoMM DiffusionDB MS COCO HPDv2 Pick-a-Pic-v2
Benchmarks: InterMT-Bench
Metrics: Pearson correlation accuracy crucial step recognition score (0-5)
Baselines: GPT-4.1 GPT-4o Gemini-2.0-Flash Gemini-2.5-Pro-preview Claude-3.7-Sonnet (thinking) o4-mini
Builds on: Bradley-Terry model RLHF-V MMDU ConvBench MMMT-IF
Closest papers by meaning
- HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses Through Reasoning MLLMs (AAAI 2026 · similarity 0.89)
- MM-RLHF: The Next Step Forward in Multimodal LLM Alignment (ICML 2025 · similarity 0.88)
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark (ICML 2024 · similarity 0.88)
- DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback (CVPR 2024 · similarity 0.88)
- PersonaVLM: Long-Term Personalized Multimodal LLMs (CVPR 2026 · similarity 0.87)
- Aligning Spoken Dialogue Models from User Interactions (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.