Research map / Vision-language and multimodal / Multimodal LLMs
AdViP: Aligning Multi-modal LLMs via Adaptive Vision-enhanced Preference Optimization
Jinda Lu, Jinghan Li, Yuan Gao, Junkang Wu, Jiancan Wu, Xiang Wang, Xiangnan He
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add vision-based preference with equal weight to DPO (ablation) | LLaVA-1.5 7B DPO · AMBER discriminative | accuracy: 64.8 -> 51.8 | Hurt |
| add vision-based preference with equal weight to DPO (ablation) | LLaVA-1.5 7B DPO · AMBER discriminative | F1: 78.8 -> 72.6 | Hurt |
| replace equal weighting with the adaptive strategy (ablation) | LLaVA-1.5 7B · AMBER discriminative | F1: 72.6 -> 85.8 | Improved |
| replace equal weighting with the adaptive strategy (ablation) | LLaVA-1.5 7B · AMBER discriminative | accuracy: 51.8 -> 79.9 | Improved |
| replace equal weighting with the adaptive strategy (ablation) | LLaVA-1.5 7B · AMBER discriminative | F1 relation: 47.4 -> 67.0 | Improved |
| apply AdaViP instead of vanilla DPO | LLaVA-1.5 7B · Object HalBench | non-hallucination rate (response): 80.1 -> 93.7 | Improved |
| apply AdaViP instead of vanilla DPO | LLaVA-1.5 13B · Object HalBench | non-hallucination rate (response): 82.3 -> 95.4 | Improved |
| apply AdaViP instead of vanilla DPO | LLaVA-1.5 7B · AMBER generative | CHAIR: 3.5 -> 1.8 | Improved |
| apply AdaViP instead of vanilla DPO | LLaVA-1.5 7B · MMHal-Bench | score: 2.00 -> 3.12 | Improved |
| apply AdaViP instead of vanilla DPO | LLaVA-1.5 13B · MMHal-Bench | hallucination rate: 30.0 -> 34.0 | Hurt |
| apply AdaViP instead of vanilla DPO | LLaVA-1.5 7B · AMBER discriminative | accuracy: 64.8 -> 79.9 | Improved |
Closest papers by meaning
- MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models (ICLR 2025 · similarity 0.92)
- CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs (ICLR 2025 · similarity 0.91)
- Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment (ICLR 2025 · similarity 0.90)
- Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
- Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment (CVPR 2025 · similarity 0.90)
- RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
