Research map / Vision-language and multimodal / Multimodal LLMs

Automated Multi-level Preference for MLLMs

NeurIPS 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AMP, a framework that builds human-free multi-level (K-rank) preference datasets for MLLMs via multi-size expert and incremental generation plus a CLIP-based auto-check, and trains them with a multi-level DPO variant (MDPO) to reduce hallucinations.

Problem

Hallucinations in multimodal LLMs; existing RLHF/DPO methods use binary superior-vs-inferior preferences that are costly to annotate (human or GPT-4V) and noisy, and are argued to be insufficient for fine-grained preference learning.

Approach

Replace binary preferences with multi-level preferences that narrow adjacent-level gaps and add cross-level comparisons, generate those preferences automatically (multi-size experts, incremental training, CLIP noun-chunk auto-check), and optimize with MDPO, a sum of pairwise DPO losses plus a penalty on the best response.

Key results

• AMP-MEG 13B reaches 3.23 MMHal-Bench score and 0.34 hallucination rate vs. 2.66/0.52 for RLHF-V and 2.01/0.61 for LLaVA-v1.5 7B.
• AMP-MEG 13B achieves the best overall POPE F1 of 87.2, above RLHF-V (85.4) and POVID (85.8).
• Using auto-check instead of GPT-4V AI annotation raises MMHal score from 2.87 to 3.17 on the MEG dataset (7B).

Measured effects reported in the paper

Surprising findings

• A 4-level preference setting outperforms 5-level, which the authors attribute to increased hidden noise.
• Comparing A>C alone performed worse than B>C alone in the adjacent-gap ablation, suggesting reducing the adjacent gap can matter more than improving the superior response.

Limitations

• Standard responses come partly from language-model superior outputs that may contain imperceptible hallucinations and can be less helpful.
• Preference learning may bias the model and reduce generalization to other capabilities.
• Multi-level preference quality degrades beyond 4 levels due to hidden noise.

Experimental setup

Datasets: ShareGPT4V Flickr30k VQAv2 LRV POPE MMHal-Bench LLaVA-Bench

Benchmarks: POPE MMHal-Bench LLaVA-Bench MRHal-Bench

Metrics: F1 accuracy MMHal score hallucination rate relative score cumulative/mean score

Baselines: LLaVA-v1.5 LLaVA-RLHF RLHF-V POVID SILKIE FGAIF Qwen-VL-Chat InstructBLIP LLaVA-13B

Builds on: DPO LLaVA-v1.5 CLIP

Closest papers by meaning

• CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs (ICLR 2025 · similarity 0.94)
• Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization (NeurIPS 2025 · similarity 0.93)
• IRIS: Implicit Reward-Guided Internal Sifting for Mitigating Multimodal Hallucination (ICML 2026 · similarity 0.92)
• Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models (CVPR 2026 · similarity 0.92)
• Stop Learning it all to Mitigate Visual Hallucination, Focus on the Hallucination Target. (CVPR 2025 · similarity 0.92)
• RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback (CVPR 2024 · similarity 0.91)