Research map / Vision-language and multimodal / Multimodal LLMs

Seeing Clearly, Reasoning Confidently: Plug-and-Play Remedies for Vision Language Model Blindness

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Bias-aware multi-image robustness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a frozen-VLM plug-and-play module that learns multi-modal class embeddings for rare objects, uses them to refine visual tokens via cross-attention and to inject detected object hints into prompts, improving rare-object reasoning on CODA-LM and GeoBench-VLM.

Problem

VLMs perform poorly at recognizing and reasoning about rare object categories due to their scarcity in pretraining data, and existing remedies require costly VLM finetuning or retrieval over large data.

Approach

Learn multi-modal class embeddings for rare objects by fusing frozen vision foundation model features with LLM-generated synonym and attribute text descriptions; use them both to refine frozen VLM visual tokens through a lightweight cross-attention adapter and as cosine-similarity detectors whose top-k outputs are injected as hints into the text prompt.

Key results

• On CODA-LM, the method raises LLaVA-1.5-7B's overall GPT score from 46.5 to 72.8 (+26.3) and Qwen2.5-VL-7B from 67.9 to 75.4, with frozen backbones.
• On GeoBench-VLM, InternVL3-8B with the module reaches 48.2 overall vs. 37.4 baseline, while LLaVA-1.5-7B + Ours (33.2) is slightly below LoRA-finetuned LLaVA-1.5-7B (34.7).
• Ablation on CODA-LM for LLaVA-1.5-7B: visual enhancement alone gives 70.2, top-k detected text hints alone give 55.8, and the full method gives 72.8.

Surprising findings

• Injecting all available class labels as hints hurts some categories (e.g., Sign drops from 48.8 to 33.1), while top-k detected hints help.
• On GeoBench-VLM, the module-equipped LLaVA-1.5-7B is slightly worse than a LoRA-finetuned LLaVA-1.5-7B (33.2 vs 34.7).

Limitations

• Gains on GeoBench-VLM for LLaVA-1.5-7B are modest, attributed to very scarce training data (361 VQA pairs).
• Evaluation limited to two benchmarks with GPT-score judging.
• Top-k detection accuracy and class-embedding design depend on LLM-generated descriptions and class-level supervision.

Experimental setup

Datasets: CODA-LM GeoBench-VLM

Benchmarks: CODA-LM GeoBench-VLM

Metrics: GPT score

Baselines: LLaVA-1.5-7B LLaVA-1.5-13B Qwen2.5-VL-7B InternVL3-8B CODA-LM MiniDrive MPDrive Jiang et al. ControlMLLM++

Builds on: CLIP LLaVA logit lens

Closest papers by meaning

• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.91)
• UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling (NeurIPS 2024 · similarity 0.90)
• Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (CVPR 2025 · similarity 0.90)
• Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.90)
• Same or Not? Enhancing Visual Perception in Vision-Language Models (CVPR 2026 · similarity 0.90)
• Towards Interpreting Visual Information Processing in Vision-Language Models (ICLR 2025 · similarity 0.89)