Research map / Vision-language and multimodal / Multimodal LLMs
Cat-PO: Cross-modal Adaptive Token-rewards for Preference Optimization in Truthful Multimodal LLMs
Zhixiao Zheng, Zheren Fu, Zhiyuan Yao, Dongming Zhang, Zhendong Mao
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Cat-PO, a cross-modal adaptive token-reward preference optimization method to reduce hallucinations in multimodal LLMs.
Problem
MLLMs hallucinate, producing text semantically inconsistent with input images; existing multimodal preference optimization methods show shortcomings at the preference data decoding stage, where different response tokens contribute differently.
Approach
Assigns adaptive token-level rewards that account for cross-modal (image-text) relevance of response tokens within preference optimization, rather than treating all tokens uniformly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add cross-modal token rewards and KL loss to DPO (Cat-PO) | LLaVA-v1.5-7B · MM-Hal | hallucination rate (%): 58.3 -> 42.0 (DPO to Cat-PO) | Improved |
| add cross-modal token rewards and KL loss to DPO | LLaVA-v1.5-7B · AMBER-Generation | CHAIR: 5.7 -> 4.8 | Improved |
| replace DPO with Cat-PO | LLaVA-v1.5-7B · AMBER-Discrimination | Accuracy: 77.5 -> 78.0 | No clear effect |
| replace DPO with Cat-PO | LLaVA-v1.5-13B · MM-Hal | Score: 2.47 -> 2.85 | Improved |
| use only attention-based token weights (ablation) | Cat-PO · MM-Hal | hallucination rate (%): 58.3 -> 55.0 (vs DPO-only) | Improved |
| use only semantic-similarity token weights (ablation) | Cat-PO · MM-Hal | hallucination rate (%): 58.3 -> 47.0 (vs DPO-only) | Improved |
| combine attention and similarity weights vs similarity only (ablation) | Cat-PO · MM-Hal | hallucination rate (%): 47.0 -> 42.0 (with KL) | Improved |
| add token-level KL term (ablation) | Cat-PO · MM-Hal | hallucination rate (%): 53.0 -> 42.0 | Improved |
| weight top 30% tokens instead of bottom 30% (ablation) | Cat-PO · MM-Hal | MM-Hal score and rate | Improved |
| make fusion weights learnable (ablation) | Cat-PO · MM-Hal | Score: 2.74 -> 2.55; rate 42% -> 50% | Hurt |
| evaluate on POPE adversarial subset versus average | Cat-PO · POPE | F1: 84.0 -> 82.5 (-2%) | No clear effect |
Experimental setup
Builds on: direct preference optimization
Closest papers by meaning
- CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs (ICLR 2025 · similarity 0.93)
- MM-RLHF: The Next Step Forward in Multimodal LLM Alignment (ICML 2025 · similarity 0.91)
- Automated Multi-level Preference for MLLMs (NeurIPS 2024 · similarity 0.90)
- Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
- Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimization (CVPR 2025 · similarity 0.89)
- Systematic Reward Gap Optimization for Mitigating VLM Hallucinations (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
