Research map / Vision-language and multimodal / Multimodal LLMs
Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FiSAO, a self-alignment method that uses a VLLM's own CLIP-style vision encoder to give token-level rewards for object tokens and optimizes with PPO (LoRA), without extra data.
Problem
Vision-language large models suffer from modality misalignment, causing hallucinations and unsafe outputs; existing preference tuning relies on coarse sentence-level rewards and external data or annotations.
Approach
Use the VLLM's own vision encoder as a fine-grained verifier: score each object token by dot product with the image embedding, set margin-based boundaries from hallucinated vs. correct object score means, and apply token-level clipped PPO rewards, requiring no reward model or extra data.
Key results
- On LLaVA-1.5, FiSAO reaches CHAIR_I 9.9 and CHAIR_S 39.9 versus 11.3 and 54.3 for the base model, and beats POVID on Cap_val (61.2 vs 57.3) and SQA-I (69.3 vs 68.8).
- On InstructBLIP, FiSAO lowers CHAIR_S from 60.3 to 42.2 and CHAIR_I from 11.9 to 8.8, and gains on MME-P (1305.3 vs 1237.5).
- The paper reports an average 8.7% improvement over existing preference-tuning approaches on the LLaVA backbone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add fine-grained rewards while using PPO (ablation) | LLaVA-1.5 · MME-P | score: 1509.3 -> 1522.6 | Helped |
| add fine-grained rewards while using PPO (ablation) | LLaVA-1.5 · Cap_val | score: 56.6 -> 61.2 | Helped |
| add fine-grained rewards while using PPO (ablation) | LLaVA-1.5 · CHAIR-I | score: 11.4 -> 9.9 | Helped |
| add PPO without fine-grained rewards (ablation) | LLaVA-1.5 · MME-P | score: 1431.9 -> 1509.3 | Helped |
| add PPO without fine-grained rewards (ablation) | LLaVA-1.5 · Cap_val | score: 54.5 -> 56.6 | Helped |
| add PPO without fine-grained rewards (ablation) | LLaVA-1.5 · CHAIR-I | score: 11.0 -> 11.4 | Hurt |
| set reward margin to 5 instead of 10 (ablation) | FiSAO on LLaVA-1.5 · Cap_val | score: 61.2 -> 57.1 | Hurt |
| set reward margin to 5 instead of 10 (ablation) | FiSAO on LLaVA-1.5 · CHAIR-I | score: 9.9 -> 10.8 | Hurt |
| set reward margin to 20 instead of 10 (ablation) | FiSAO on LLaVA-1.5 · MME-P | score: 1522.6 -> 1501.4 | Hurt |
| set reward margin to 20 instead of 10 (ablation) | FiSAO on LLaVA-1.5 · CHAIR-I | score: 9.9 -> 13.6 | Hurt |
| apply FiSAO instead of POVID | LLaVA-1.5 · Cap_val | score: 57.3 -> 61.2 | Helped |
| apply FiSAO instead of POVID | InstructBLIP · CHAIR-I | score: 10.5 -> 8.8 | Helped |
Surprising findings
- Sentence-level CLIP scores show near-zero correlation with BLEU and ROUGE (r = -0.01), so coarse rewards are weak signals.
- On POPE, FiSAO (85.7) does not beat the base LLaVA-1.5 (85.9), and on MM-Vet it matches the base model at 30.5.
Limitations
- Performance depends on the quality and biases of the pre-trained vision encoder, which propagates into the reward.
- Rewards are restricted to a predefined set of common object entities (about 5678 words), so coverage is limited.
- Evaluated only on two backbones (LLaVA-1.5 7B and InstructBLIP) and mostly with LoRA.
Experimental setup
Datasets: LLaVA-Instruct 150k COCO Detic labels CHAIR
Benchmarks: MME SEED-Bench MMBench MM-Vet ScienceQA POPE GQA COCO caption CHAIR
Metrics: MME perception and cognition scores accuracy CHAIR_S CHAIR_I BLEU ROUGE-L CIDEr
Baselines: Silkie (VLFeedback) LLaVA-RLHF POVID BLIP-2 Qwen-VL-Chat mPLUG-Owl2
Builds on: PPO DPO LLaVA-1.5 InstructBLIP CHAIR
Closest papers by meaning
- Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.91)
- Calibrated Self-Rewarding Vision Language Models (NeurIPS 2024 · similarity 0.91)
- Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization (NeurIPS 2025 · similarity 0.90)
- MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models (ICLR 2025 · similarity 0.90)
- Deep Pre-Alignment for VLMs (ICML 2026 · similarity 0.90)
- SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Models (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.