Research map / Vision-language and multimodal / Multimodal LLMs
Anchor-Final Self-Supervision Drives Hallucination-Aware Optimization in Large Vision-Language Models
Jiaxi Liu, Yifeng Yang, Xinbing Wang, Qinying Gu, Nanyang Ye
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AFS (Anchor-Final Self-Supervision), a framework that uses discrepancies between intermediate-layer and final-layer predictions of an LVLM to selectively apply self-supervision for hallucination-aware optimization.
Problem
Large vision-language models often generate tokens that are not aligned with visual evidence (hallucinations).
Approach
Uses the disagreement between intermediate ("anchor") and final layer predictions as a self-supervision signal, applied selectively to visual tokens, to optimize the model against hallucinations without external annotations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add visual-selective gating (ablation) | AFS reward · CHAIR | CHAIR_S: 26.2 -> 24.6 | Improved |
| add hallucination-aware token classification (ablation) | AFS reward · CHAIR | CHAIR_S: 27.2 -> 24.6 | Improved |
| add hallucination-aware token classification (ablation) | AFS reward · CHAIR | CHAIR_I: 8.8 -> 7.7 | Improved |
| add anchor-final distributional consistency (ablation) | AFS reward · CHAIR | CHAIR_S: 25.6 -> 24.6 | Improved |
| add anchor-final distributional consistency (ablation) | AFS reward · CHAIR | Recall: 63.9 -> 64.6 | Improved |
| add token-specific weight (ablation) | AFS reward · CHAIR | CHAIR_S: 25.8 -> 24.6 | Improved |
| add hallucination-aware token classification (ablation) | AFS reward · CHAIR | Recall: 65.7 -> 64.6 | Hurt |
| use anchor layers 20-27 instead of 14-21 (ablation) | AFS · CHAIR | CHAIR_S: 25.8 -> 24.6 | Improved |
| use anchor layers 20-27 instead of 17-24 (ablation) | AFS · CHAIR | CHAIR_S: 25.6 -> 24.6 | Improved |
| set reward sharpness alpha 1.0 instead of 0.5 (ablation) | AFS · CHAIR | CHAIR_S: 25.0 -> 24.6 | Improved |
| set reward sharpness alpha 1.0 instead of 2.0 (ablation) | AFS · CHAIR | CHAIR_S: 25.4 -> 24.6 | Improved |
| train with AFS self-supervised reward instead of standard GRPO with binary GQA reward | Qwen2.5-VL-7B · CHAIR | CHAIR_S: 29.4 -> 24.6 | Improved |
Closest papers by meaning
- Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.93)
- Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models (ICLR 2025 · similarity 0.93)
- Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models (ICLR 2025 · similarity 0.93)
- Locate-then-Sparsify: Attribution Guided Sparse Strategy for Visual Hallucination Mitigation (CVPR 2026 · similarity 0.93)
- Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.93)
- Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback (AAAI 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
