Research map / Vision-language and multimodal / Multimodal LLMs
Controlling Multimodal LLMs via Reward-guided Decoding
Oscar Mañas, Pierluca D'Oro, Koustuv Sinha, Adriana Romero-Soriano, Michal Drozdzal, Aishwarya Agrawal
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add reward-guided decoding with hallucination reward (w=1.0) over greedy | LLaVA-1.5-7B · COCO captions | CHAIR_i: 15.05 -> 4.53 | Improved |
| use hallucination reward only (w=1.0) instead of greedy, recall cost | LLaVA-1.5-7B · COCO captions | object recall: 81.30 -> 76.04 | Hurt |
| mix recall reward (w=0.5) instead of w=1.0 (ablation) | MRGD LLaVA-1.5-7B · AMBER | coverage: 52.4 -> 60.8 | Improved |
| use recall reward only (w=0.0) instead of greedy | LLaVA-1.5-7B · COCO captions | recall and CHAIR_i: recall 81.30 -> 85.23; CHAIR_i 15.05 -> 24.20 | Mixed |
| use MRGD instead of CGD | LLaVA-1.5-7B · COCO captions | CHAIR_i: 9.48 -> 4.53 | Improved |
| fine-tune PaliGemma on preference data instead of off-the-shelf SigLIP for r_hal (ablation) | MRGD LLaVA-1.5-7B · COCO captions | CHAIR_i: 7.19 -> 4.53 | Improved |
| add RLAIF-V to the r_hal preference data (ablation) | MRGD LLaVA-1.5-7B · COCO captions | CHAIR_i: 5.34 -> 7.83 | Hurt |
| add RLAIF-V and remove POVID from r_hal data (ablation) | MRGD LLaVA-1.5-7B · COCO captions | CHAIR_i: 5.34 -> 8.17 | Hurt |
| use PaliGemma-2 backbone instead of PaliGemma for r_hal (ablation) | MRGD LLaVA-1.5-7B · COCO captions | CHAIR_i: 5.34 -> 5.88 | No clear effect |
| use DETR instead of OWLv2 object detector for r_rec (ablation) | MRGD LLaVA-1.5-7B · COCO captions | CHAIR_i: 5.34 -> 5.37 | No clear effect |
| evaluate rewards every sentence (T=1) with k=5 instead of rejection sampling with k=30 (ablation) | MRGD LLaVA-1.5-7B · COCO captions | CHAIR_i: k=5 beats rejection sampling k=30; over 6x more sample-efficient | Improved |
| add prompting for accuracy | Llama-3.2-Vision-11B · COCO captions | CHAIR_s: 20.52 -> 25.24 | Hurt |
Closest papers by meaning
- Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation (ICLR 2026 · similarity 0.90)
- First Logit Boosting: Visual Grounding Method to Mitigate Object Hallucination in Large Vision-Language Models (CVPR 2026 · similarity 0.90)
- PostAlign: Multimodal Grounding as a Corrective Lens for MLLMs (ICLR 2026 · similarity 0.90)
- Breaking the Illusion: When Positive Meets Negative in Multimodal Decoding (CVPR 2026 · similarity 0.90)
- MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation (ICLR 2025 · similarity 0.90)
- Calibrated Self-Rewarding Vision Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
