Research map / Vision-language and multimodal / Multimodal LLMs
CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CHiP, a DPO extension for MLLMs that adds visual preference pairs (perturbed images as rejected inputs) and hierarchical response/segment/token-level textual preference terms to reduce hallucinations in LLaVA-1.6 and Muffin.
Problem
Multimodal DPO built from text-only preference pairs fails to align image and text representations and to separate hallucinated from non-hallucinated descriptions, leaving MLLM hallucinations under-addressed.
Approach
Combines a visual preference term, where the chosen response is scored against a perturbed image (diffusion-noised, cropped, rotated, etc.) as the rejected image, with a hierarchical textual DPO that adds segment-level reweighting of changed spans and a sequential-KL token-level term on top of standard response-level DPO.
Key results
- On Object HalBench, LLaVA-1.6 response-level hallucination drops from 11.0 (DPO) to 4.9 (CHiP), and Muffin from 13.1 to 6.2.
- On AMBER, CHiP reduces LLaVA-1.6 hallucination rate (Hal) from 38.9 (DPO) to 24.5, and Muffin from 26.5 to 17.6.
- General capability on LLaVA-Wild rises from 74.9 to 76.2 and MMMU(val) from 35.8 to 36.8 with CHiP, while MMB-ENG drops 0.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CHiP instead of DPO | Muffin (13B) · ObjHal | response hallucination rate: 13.1 -> 6.2 | Helped |
| Apply CHiP instead of DPO | LLaVA-1.6 (7B) · ObjHal | response hallucination rate: 11.0 -> 4.9 | Helped |
| Apply CHiP instead of DPO | Muffin (13B) · MMHal | overall score: 2.5 -> 2.6 | Helped |
| Apply CHiP instead of DPO | LLaVA-1.6 (7B) · MMHal | overall score: 2.7 -> 2.9 | Helped |
| Apply CHiP instead of DPO | Muffin (13B) · HallusionBench | All Accuracy: 52.5 -> 54.0 | Helped |
| Apply CHiP instead of DPO | LLaVA-1.6 (7B) · HallusionBench | All Accuracy: 56.6 -> 58.5 | Helped |
| Apply CHiP instead of DPO | Muffin (13B) · AMBER | hallucination rate (Hal): 26.5 -> 17.6 | Helped |
| Apply CHiP instead of DPO | LLaVA-1.6 (7B) · AMBER | hallucination rate (Hal): 38.9 -> 24.5 | Helped |
| Add visual preference optimization (ablation) | CHiP on LLaVA-1.6 (7B) · ObjHal | response hallucination rate: 4.92 -> 9.19 | Helped |
| Add segment-level preference optimization (ablation) | CHiP on LLaVA-1.6 (7B) · ObjHal | response hallucination rate: 4.92 -> 8.55 | Helped |
| Add token-level preference optimization (ablation) | CHiP on LLaVA-1.6 (7B) · ObjHal | response hallucination rate: 4.92 -> 6.08 | Helped |
| Add segment- and token-level preference optimization (ablation) | CMDPO on LLaVA-1.6 (7B) · ObjHal | response hallucination rate: 9.76 -> 4.92 | Helped |
Surprising findings
- Training the visual encoder with CHiP does not reduce hallucinations further, unlike DPO, which the authors attribute to diluted focus on image-text alignment.
- Token-level preference optimization alone (TDPO) outperforms DPO on hallucination benchmarks.
Limitations
- Gains on HallusionBench fA are lower than DPO for LLaVA
- Joint training of the visual encoder weakens CHiP's benefit
- Evaluation partly relies on GPT-4 judges with incomplete MMHal annotations, supplemented by a small human study
Experimental setup
Datasets: RLHF-V-Dataset COCO-2017
Benchmarks: Object HalBench MMHal-Bench HallusionBench AMBER MMMU ScienceQA LLaVA-Wild
Metrics: response-level hallucination rate mention-level hallucination rate overall score accuracy CHAIR coverage
Baselines: DPO LLaVA-1.6 Muffin GPT-4V RLHF-V HALVA LLaVA-RLHF InstructBLIP
Builds on: DPO TDPO RLHF-V LLaVA-1.6 Muffin
Closest papers by meaning
- Automated Multi-level Preference for MLLMs (NeurIPS 2024 · similarity 0.94)
- Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization (NeurIPS 2025 · similarity 0.93)
- Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models (CVPR 2026 · similarity 0.93)
- Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key (CVPR 2025 · similarity 0.92)
- MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization (CVPR 2026 · similarity 0.92)
- Mitigating Visual Hallucinations via Semantic Curriculum Preference Optimization in MLLMs (ICML 2026 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.