Research map / Vision-language and multimodal / Visual question answering
Towards Rationale-Answer Alignment of LVLMs via Self-Rationale Calibration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Self-Rationale Calibration (SRC), an iterative framework that fine-tunes LVLMs to output rationale-then-answer, samples sentence-level beam candidates, ranks them with a pairwise LLM scorer (R-Scorer, 1.5B) plus confidence weighting, and calibrates via DPO.
Problem
LVLMs often produce answers whose rationales are counterfactual, insufficient, or unreasonable, so the rationale and answer are misaligned even when the answer is correct; existing vision-text preference alignment focuses on captions rather than VQA rationales.
Approach
Use the model's own sentence-level beam-search candidates of rationale-answer pairs, score them pairwise with a small distilled LLM judge that includes factual checks, combine rank of winning score with model confidence, and run iterative DPO on best vs. worst candidates.
Key results
- SRC raises LLaVA-1.5-7B MMStar overall from 32.1 to 38.9 and LLaVA-Next-8B from 42.6 to 47.6.
- On HallusionBench, LLaVA-Next-8B improves from 32.1 to 37.5; on ChartQA from 68.7 to 71.4.
- The 1.5B R-Scorer achieves a 81.6% win rate over the original Qwen-2.5-1.5B and 57.6% over LLaMA-3.1-70B in human alignment, and 42.0% vs GPT-4o.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Self-Rationale Calibration | LLaVA-1.5-7B · MMStar Overall | score: 32.1 to 38.9 | Helped |
| Apply Self-Rationale Calibration | LLaVA-Next-8B · MMStar Overall | score: 42.6 to 47.6 | Helped |
| Apply Self-Rationale Calibration | LLaVA-1.5-7B · LLaVA-W | score: 66.3 to 65.5 | Hurt |
| Remove rationale fine-tuning (ablation) | SRC on LLaVA-1.5-7B · MMStar Overall | score: 38.9 to 34.0 | Hurt |
| Remove rationale fine-tuning (ablation) | SRC on LLaVA-Next-8B · MMStar Overall | score: 47.6 to 44.2 | Hurt |
| Use count instead of sum for the winning score (ablation) | SRC scoring on LLaVA-1.5-7B seed · MMStar Overall | score: 36.5 to 35.4 | Hurt |
| Add factual checks during candidate scoring (ablation) | SRC scoring on LLaVA-1.5-7B seed · MMStar Fine-grained Perception | score: 26.4 to 33.6 | Helped |
| Use pairwise rather than independent scoring (ablation) | SRC with Qwen-2.5-72B scoring · MMStar Fine-grained Perception | score: +6.4 | Helped |
| Increase R-Scorer size from 1.5B to 7B (ablation) | SRC scoring on LLaVA-1.5-7B seed · MMStar Overall | score: 36.5 to 36.8 | Helped |
| Use natural-language Thinking and Output format rather than <thinking>/<output> tags (ablation) | Rationale fine-tuning on LLaVA-1.5-7B · MMStar Overall | score: 29.4 to 34.7 | Helped |
| Use natural-language Thinking and Output format rather than [thinking]/[output] tags (ablation) | Rationale fine-tuning on LLaVA-1.5-7B · MMStar Overall | score: 31.3 to 34.7 | Helped |
| Use rationale SFT rather than SFT without rationale (ablation) | LLaVA-1.5-7B · MMStar Overall | score: 33.7 to 34.7 | Helped |
Surprising findings
- Explicit special tags such as <thinking>/<output> degraded performance after rationale fine-tuning, while natural-language tags worked best.
- Prompting LLaVA-Next for step-by-step rationales without fine-tuning lowered overall MMStar score (42.6 to 40.8) despite gains in math.
Limitations
- Scoring relies on GPT-4o-labeled pairs and manual filtering, and the R-Scorer was trained without factual checks.
- Gains on LLaVA-Bench were a minor shortfall versus DPO baselines, attributed to format changes affecting LLM-as-judge evaluation.
- Low confidence does not necessarily imply a non-preferred candidate, so confidence is a noisy signal.
Experimental setup
Datasets: LLaVA-150k VQAv2 ShareGPT-4V GQA IDK TallyQA VizWiz OODVQA ChartQA DocVQA MathVision AI2D
Benchmarks: MMStar ChartQA LLaVA-Bench HallusionBench
Metrics: accuracy win rate semantic entropy
Baselines: POVID HA-DPO SIMA SeVA RLAIF-V CSR Qwen-2.5-72B scorer LLaMA-3.1-70B scorer GPT-4o scorer
Builds on: Direct Preference Optimization LLaVA-1.5 RPO LLM-as-judge
Closest papers by meaning
- Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (ICLR 2024 · similarity 0.88)
- Unveiling the Tapestry of Consistency in Large Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger (ICML 2025 · similarity 0.88)
- Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization (NeurIPS 2025 · similarity 0.88)
- Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation (CVPR 2025 · similarity 0.87)
- UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.