Research map / Vision-language and multimodal / Multimodal LLMs

LVLM-Aided Alignment of Task-Specific Vision Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LVLM-VA, which uses a GPT-4o Critic and Judge on XAI segmentations (PPEPS-WGM) to turn class-level human descriptions into instance-wise correction masks that fine-tune small vision models via the RRR loss.

Problem

Small task-specific vision models rely on spurious correlations rather than domain-aligned features, causing brittle real-world behavior, and existing alignment methods need costly instance-level feedback or group labels.

Approach

An LVLM acts as a bidirectional translator: it judges whether explanation-map clusters (from a weighted Gaussian mixture over positive attribution mass) are spurious, given class-level human descriptions, and the resulting binary verdicts form correction masks used in RRR-loss fine-tuning.

Key results

• On knee radiographs, LVLM-VA improves worst group accuracy by 0.16 ± 0.06 (7 seeds) while maintaining overall accuracy; SUBG gains more WGA (0.3457 absolute WGA vs. 0.1914 for LVLM-VA) but at a large AGA cost.
• On skin lesions, LVLM-VA reaches WGA 0.4457 ± 0.0894 vs. 0.2300 original, with AGA 0.7493 vs. 0.7400 original.
• User study with 18 participants: 88% agreement with cluster relevance, 87% with critic reasoning, 86% preferring post-alignment explanations.

Surprising findings

• SUBG achieves higher worst group accuracy than LVLM-VA on the knee dataset but sacrifices substantial overall accuracy.
• GPT-4o-mini and Qwen3 models achieve much lower verdict accuracy (0.42 and 0.46) than GPT-4o, GPT-5 and Llama-4.

Limitations

• Relies on class-level human descriptions that may be hard for experts to formalize.
• Distinction between core and spurious features is not always clear or spatially separable.
• Verdict quality depends strongly on the LVLM; weaker models perform poorly.

Experimental setup

Datasets: DecoyMNIST ISIC skin lesion dataset knee osteoarthritis radiograph dataset MedMNIST-C

Benchmarks: DecoyMNIST ISIC skin lesion knee osteoarthritis radiographs MedMNIST-C

Metrics: worst group accuracy average group accuracy verdict accuracy alignment metric (attribution mass outside spurious region) user study agreement

Baselines: SUBG DFR JTT SAM-based segmentation ground-truth expert masks RRR

Builds on: Right for the Right Reasons (RRR) DeepLIFTSHAP Segment Anything Model Deep Feature Reweighting Just Train Twice

Closest papers by meaning

• Beyond Sight: Towards Cognitive Alignment in LVLM via Enriched Visual Knowledge (CVPR 2025 · similarity 0.91)
• Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.90)
• Vision-Language Models Create Cross-Modal Task Representations (ICML 2025 · similarity 0.90)
• SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Models (CVPR 2025 · similarity 0.90)
• VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set (NeurIPS 2025 · similarity 0.89)
• MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs (ICML 2026 · similarity 0.89)