Research map / Vision-language and multimodal / Multimodal LLMs

Mitigating Visual Hallucinations via Semantic Curriculum Preference Optimization in MLLMs

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Preference tuning against hallucination · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SCPO, a curriculum DPO variant for MLLMs that uses difficulty-sorted semantic preference pairs (SCPP), a symmetric bidirectional loss (CCO+CSO), and a dynamic reference model reset per stage to reduce visual hallucinations in LLaVA models.

Problem

Visual hallucinations in MLLMs persist; standard DPO-style alignment with random image perturbations captures low-level distortions rather than fine-grained semantics, is one-sided toward the image, and uses static single-pass training with a fixed reference model.

Approach

Combines a semantically hard, difficulty-ranked preference dataset with a symmetric text-and-image preference objective (CCO plus CSO) and a staged curriculum that resets the DPO reference model to the previous stage's policy at each stage.

Key results

• SCPO-7B on LLaVA-v1.6-7B reduces Object HalBench CHAIRs from 12.0 to 7.0 and AMBER HalRate from 49.7 to 27.1.
• On LLaVA-v1.5-7B, the Hard-stage SCPO model reaches CHAIRs 20.3 vs 54.7 for the base model, and MMHal Score 2.47 vs 2.18.
• SCPO-7B achieves a 48.5% relative CHAIRs reduction versus GPT-4V (7.0 vs 13.6).

Surprising findings

• Sorting data by difficulty alone and dynamic reference updates alone each give only moderate gains, while reversing the curriculum order makes Object HalBench CHAIRs worsen across stages.
• The LLaVA-v1.6-7B Easy stage lowers LLaVA-wild score from 71.4 to 67.5, showing general-capability trade-offs in some stages.

Limitations

• Preference data partly generated by GPT-5 and the difficulty score relies on MLLM entropy, CLIP and DINOv2 features, which may introduce bias.
• Datasets and models are promised for release only after review; no release link given.
• Some baseline numbers are cited from other papers rather than re-run under identical settings; hyperparameter sweep is limited to LLaVA-v1.6-7B.

Experimental setup

Datasets: RLHF-V MVC HumanEdit Winoground SCPP

Benchmarks: Object HalBench AMBER-Generative AMBER-Discriminative MMHal Bench LLaVA-in-the-Wild MMBench-CN

Metrics: CHAIRs CHAIRi CHAIR Cover HalRate Cog F1-Gen. Acc. F1-Dis. Score

Baselines: GPT-4V Qwen-VL-Chat Silkie InstructBLIP MiniGemini DPO mDPO EOS HA-DPO HSA-DPO RLHF-V (HD) HALVA

Builds on: DPO RLHF-V mDPO HA-DPO EOS

Closest papers by meaning

• Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key (CVPR 2025 · similarity 0.93)
• Systematic Reward Gap Optimization for Mitigating VLM Hallucinations (NeurIPS 2025 · similarity 0.92)
• CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs (ICLR 2025 · similarity 0.92)
• Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models (CVPR 2026 · similarity 0.92)
• Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization (NeurIPS 2025 · similarity 0.91)
• Learning from Fine-Grained Visual Discrepancies: Mitigating Multimodal Hallucinations via In-Context Visual Contrastive Optimization (ICML 2026 · similarity 0.91)