Research map / Vision-language and multimodal / Multimodal LLMs

PostAlign: Multimodal Grounding as a Corrective Lens for MLLMs

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MMGrounded-PostAlign, which adds visual grounding (SAM-based mask and box decoding with a <REJ> rejection token) and selective textual rationale generation on top of LLaVA-1.5 to reduce hallucinations.

Problem

MLLMs over-rely on linguistic priors and spurious co-occurrences, causing object/attribute hallucinations and weak fine-grained visual understanding.

Approach

Attach a SAM-based multi-task grounding decoder to an MLLM driven by a <LOC> token, add a <REJ> token that yields empty masks for absent referents, and use a <SIMPLE>/<COMPLEX> token so the model generates rationales only for complex queries.

Key results

• On POPE, MMGrounded-PostAlign-13B reaches 88.9 (random), 87.3 (popular), 85.6 (adversarial) vs. 85.4/82.2/79.2 for the 13B baseline.
• On HaloQuest False Premise, the full model (SEG+DET+REJ) scores 33.2 human eval vs. 9.9 without REJ.
• On MMBench EN, PostAlign-7B scores 63.9 vs. 62.2 baseline; VQAv2 improves from 77.4 to 78.8.

Surprising findings

• Removing the image input still reproduces 89.2% of hallucinated tokens, which the authors use to argue hallucinations mainly stem from linguistic priors.
• BTL-Generation (directly generating boxes as text) degrades MMBench and VQAv2 reasoning performance relative to baseline.

Limitations

• Relies on grounding-augmented training data whose construction is not described in detail.
• Grounding performance is not directly optimized, and the paper does not increase grounding-specific data.
• Benchmark evaluations are partly human/auto-eval on HaloQuest with unclear judge details.

Experimental setup

Datasets: MSCOCO RefCOCO ReasonSeg

Benchmarks: POPE HaloQuest VQAv2 MME MMBench RefCOCO RefCOCO+ RefCOCOg ReasonSeg

Metrics: accuracy human evaluation score auto-eval score gIoU cIoU REC accuracy RES mIoU

Baselines: LLaVA-1.5 baseline (BTL-Generation/Caption variants) Kosmos-2 LISA-7B LLaVA-Grounding-7B VisionLLM v2 PixelLLM GLaMM GSVA-7B MiniGPT v2-7B Shikra-7B Ferret-7B Volcano

Builds on: LLaVA-1.5 SAM LISA LoRA Kosmos-2

Closest papers by meaning

• Grounded Chain-of-Thought for Multimodal Large Language Models (CVPR 2026 · similarity 0.92)
• GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.92)
• Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignment (ICLR 2025 · similarity 0.91)
• PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training (ICLR 2025 · similarity 0.91)
• GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models (AAAI 2026 · similarity 0.91)
• Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attention (CVPR 2026 · similarity 0.90)