Research map / Vision-language and multimodal / Multimodal LLMs

Can Large Vision-Language Models Correct Semantic Grounding Errors By Themselves?

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that off-the-shelf VLMs can self-correct semantic grounding errors when oracle or self-generated binary feedback is prompted with visual marks, RoI crops and zero-shot CoT, improving accuracy up to 8.4 points on ADE20k and COCO without training.

Problem

Open VLMs and API VLMs make semantic grounding errors (naming the class of image regions), and fine-tuning to fix them is costly; it is unclear whether VLMs can self-correct these errors via prompt-based feedback, which LLM work has found difficult.

Approach

Decompose self-correction into binary verification (is this region's label correct?) performed by the same VLM on an RoI-cropped or visually marked image, then iteratively refine predictions using that feedback, with no training, oracle access, or architectural change.

Key results

• VLM binary verification with RoI crop reaches F1 of 61.71 on LLaVA-1.5 versus 51.12 for intrinsic self-correction, and up to 18.81 points higher on feedback F1 across models.
• Iterative self-generated binary feedback improves grounding accuracy without oracle, e.g. LLaVA-1.5 from 35.86 to 40.29 over five rounds, while intrinsic self-correction lowers accuracy.
• With oracle binary feedback, GPT-4o rises from 33.81 to 57.78 after three rounds, yet error rates remain above 40% on ADE20k.

Measured effects reported in the paper

Surprising findings

• Intrinsic self-correction (asking the VLM to review and refine its answer) degrades accuracy for most open-source VLMs, e.g. CogVLM drops by up to 10.18 points.
• Even with oracle class-label feedback that states the ground truth, open-source VLMs fail in about 25% of cases.

Limitations

• Class mapping via sentence embeddings introduces noise; a human study found 10% of outputs hard to map to the closed label set.
• Verifier feedback can be noisy, causing correct predictions to be flipped (failure cases in COCO).
• Evaluated on only 100 images per dataset, with open-source models at 13B scale.

Experimental setup

Datasets: ADE20k panoptic segmentation (SoM split, 100 images) COCO panoptic segmentation (SoM split, 100 images)

Benchmarks: ADE20k-150 SoM split COCO panoptic SoM split

Metrics: classification accuracy F1 score for binary feedback accuracy of feedback

Baselines: Intrinsic Self-Correction Set-of-Mark prompting Zero-shot CoT

Builds on: Self-Refine Chain-of-Thought Set-of-Mark prompting CoVe intrinsic self-correction

Closest papers by meaning

• Sherlock: Self-Correcting Reasoning in Vision-Language Models (NeurIPS 2025 · similarity 0.89)
• Investigating Compositional Challenges in Vision-Language Models for Visual Grounding (CVPR 2024 · similarity 0.89)
• Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning (AAAI 2026 · similarity 0.89)
• Towards Self-Refinement of Vision-Language Models with Triangular Consistency (NeurIPS 2025 · similarity 0.89)
• GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models (AAAI 2026 · similarity 0.88)
• GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.88)