Research map / Vision-language and multimodal / Multimodal LLMs
Calibrated Self-Rewarding Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Calibrated Self-Rewarding (CSR), an iterative self-preference method for LVLMs that uses sentence-level beam search with a reward combining the model's own likelihood and a CLIP image-response relevance score, then DPO-tunes LLaVA-1.5 and Vila.
Problem
LVLMs hallucinate because they over-rely on text priors and under-use visual input; existing preference-data methods depend on human annotation or external models (e.g., GPT-4), and naive self-rewarding ignores the visual modality when generating preferences.
Approach
Calibrate the self-rewarding signal with a CLIP-based image-response relevance score, combined with the model's own sentence-level cumulative likelihood in a step-wise reward during sentence-level beam search; the highest- and lowest-reward responses form DPO pairs, iterated over several rounds.
Key results
- CSR on LLaVA-1.5-7B improves average score by about 7.62% over iterations; LLaVA-Bench rises from 63.4 to 71.1 and CHAIR_S drops from 48.8 to 21.0.
- CSR outperforms the self-rewarding baseline by an average of 2.43% on LLaVA-1.5 7B and reaches 74.7 on LLaVA-Bench with 13B.
- Applied to Vila-7B, CSR improves overall performance by 3.37% after three iterations, with MM-Vet up 14.0% and VisWiz up 8.48%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Calibrated self-rewarding (CSR) with 3 iterations vs. plain self-rewarding | LLaVA-1.5 7B · Comprehensive, VQA and hallucination benchmarks (Table 1) | average score across benchmarks: CSR outperforms self-rewarding by 2.43% on average | Helped |
| Reward combining image-response relevance (R_I) with instruction-following score (R_T) vs. R_T only (ablation) | LLaVA-1.5 7B · Vision-text reward ablation | average score: 72.39 vs 68.46 (Only R_T) | Helped |
| Reward combining R_I and R_T vs. R_I only (ablation) | LLaVA-1.5 13B · Vision-text reward ablation | average score: 71.95 vs 69.23 (Only R_I) | Helped |
| Adding image-response relevance term (R_I) to reward, 13B: vs R_T only (ablation) | LLaVA-1.5 13B · Vision-text reward ablation | average score: 68.12 (Only R_T) vs 71.95 (CSR) | Mixed |
| Increasing image-relevance weight lambda (0.1 to 0.9) (ablation) | LLaVA-1.5 7B · Benchmarks in appendix Table 11 | LLaVA^W score: 66.7 (lambda=0.1) to 71.1 (lambda=0.9) | Helped |
| Iterative CSR (iterations 1 to 3) | LLaVA-1.5 7B · Average score over benchmarks (Figure, appendix Table 7) | average score: 66.61 base to 72.24 after iter-5; iter-3 is 71.74 | Helped |
| Iterations beyond 3 (iter-4, iter-5) | LLaVA-1.5 7B · Benchmark suite (appendix Table 7/8) | average score: iter-3 71.74 vs iter-5 72.24; gains shrink | No clear effect |
| CSR applied to Vila 7B (3 iterations) | Vila 7B · Ten benchmarks | average score: overall +3.37%; VisWiz and MM-Vet up 8.48% and 14.0% | Helped |
| CSR (3 iterations) vs self-rewarding on LLaVA-1.5 13B | LLaVA-1.5 13B · LLaVA^W and CHAIR_S (Table 1) | LLaVA^W score and CHAIR_S: LLaVA^W 74.7 vs 65.6; CHAIR_S 28.0 vs 37.0 (lower better) | Helped |
| Using GPT-4-modified negative responses (external preference data) vs self-generated negatives | LLaVA-1.5 7B · Analysis of preference sources (appendix) | sampling probability distribution of dispreferred responses | Hurt |
Surprising findings
- Responses modified by GPT-4V into dispreferred ones are easier for the target LVLM to distinguish than its own generated dispreferred responses, supporting self-generated preference data.
- Chosen and rejected image-relevance scores converge over iterations, making self-generated pairs harder to distinguish while still improving learning.
Limitations
- Only three iterations were run due to compute limits, so scaling behavior is unknown.
- Experiments were limited to 7B and 13B models.
- Hallucinations still occur after CSR, as the authors note.
Experimental setup
Datasets: LLaVA-150k VLFeedback COCO val2014 (500 sampled images for CHAIR)
Benchmarks: MME SEED-Bench LLaVA-Bench (In-the-Wild) MMBench MM-Vet ScienceQA-IMG VizWiz GQA POPE CHAIR
Metrics: accuracy MME perception and cognition scores CHAIR_S CHAIR_I POPE accuracy and F1 average score over benchmarks
Baselines: Self-rewarding Silkie (VLFeedback) LLaVA-RLHF POVID RLHF-V InstructBLIP Qwen-VL-Chat mPLUG-Owl2 BLIP-2 IDEFICS
Builds on: DPO self-rewarding language models process-supervised reward models LLaVA-1.5 CLIP
Closest papers by meaning
- Systematic Reward Gap Optimization for Mitigating VLM Hallucinations (NeurIPS 2025 · similarity 0.91)
- RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data (AAAI 2025 · similarity 0.91)
- Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation (ICML 2026 · similarity 0.91)
- Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models (ICLR 2025 · similarity 0.91)
- VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models (CVPR 2025 · similarity 0.91)
- TLDR: Token-Level Detective Reward Model for Large Vision Language Models (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.