Research map / Vision-language and multimodal / Multimodal LLMs

Symbal: Detecting Systematic Misalignments in Model-Generated Captions

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces systematic misalignment detection for MLLM-generated captions, a two-stage dual-stage method (Symbal) that finds recurring erroneous textual facts and their associated visual features, plus a 420-setting benchmark built by injecting controlled misalignments into COCO and MIMIC-CXR.

Problem

Captions generated by multimodal LLMs often contain errors that recur systematically with a specific visual feature in the image, and no automated method or benchmark exists to detect such dataset-level misalignments, especially without access to the generating MLLM.

Approach

Decompose systematic misalignment detection into two stages (erroneous textual fact, then associated visual feature), each with group (spherical K-Means over embeddings), score (embedding, text-only, or vision-language alignment), and summarize steps using off-the-shelf foundation models; evaluate on a benchmark built by injecting known misalignments with controlled Cramer's V association.

Key results

• Best Symbal configuration identifies the systematic misalignment in up to 63.8% of benchmark datasets, reported as a nearly 4x improvement over the closest baseline (GPT-OSS 120B, 17.1% best case).
• Stage 1 erroneous-fact detection reaches 94.2% Acc@5 on natural reference-free settings, while Stage 2 visual-feature detection reaches 69.7% Acc@5.
• On real captions, flagged misalignments are 3.1x (handbag–bus, Llava1.5-7B), 4.6x (text–sign, LlavaOneVision-7B), and 17.2x (white tablecloth–table, ShareGPT4V) more likely when the visual feature is present.

Surprising findings

• Visual feature identification is substantially harder than textual error identification, and medical-domain visual feature detection is especially hard (36.7% Acc@5 reference-free).
• In the natural domain, reference-free settings often outperform reference-based ones, attributed to sparse COCO reference captions.

Limitations

• Benchmark uses injected synthetic misalignments in COCO and MIMIC-CXR, which may not capture real MLLM error distributions.
• Medical visual feature identification performs poorly, with 36.7% Acc@5 reference-free.
• Evaluation relies on Llama3.3-70B as LLM-as-a-judge for equivalence matching.

Experimental setup

Datasets: COCO 2017 val MIMIC-CXR test ShareGPT4V

Benchmarks: Symbal benchmark (420 settings)

Metrics: Accuracy@1 Accuracy@5

Baselines: Llama3.3 70B direct prompting Qwen2.5-VL 72B direct prompting GPT-OSS 120B direct prompting

Builds on: CLIPScore spherical K-Means

Closest papers by meaning

• Extract Free Dense Misalignment from CLIP (AAAI 2025 · similarity 0.87)
• LEMoN: Label Error Detection using Multimodal Neighbors (ICML 2025 · similarity 0.86)
• SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.86)
• VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.85)
• Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data (CVPR 2025 · similarity 0.85)
• Semantic and Expressive Variations in Image Captions Across Languages (CVPR 2025 · similarity 0.85)