Research map / Vision-language and multimodal / Multimodal LLMs

Reflexive Guidance: Improving OoDD in Vision-Language Models via Self-Guided Image-Adaptive Concept Generation

ICLR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Evaluates out-of-distribution detection (OoDD) of proprietary and open-source vision-language models via confidence scores in generated text, and proposes Reflexive Guidance (ReGuide), a two-stage self-guided prompting method where the LVLM suggests image-adaptive near/far OoD concepts used as auxiliary negative classes.

Problem

Out-of-distribution detection capabilities of large vision-language models are underexplored, especially when confidence must be expressed through natural language responses rather than logits.

Approach

Prompt the LVLM itself to suggest image-adaptive near-OoD and far-OoD class names from the input image, then use these as auxiliary OoD classes (with a 'none of these classes' rejection option) in a second OoDD prompt, using maximum ID-class confidence as the OoD score.

Key results

• On ImageNet200 with InternVL2-26B, ReGuide raises All-OoD AUROC from 63.11 to 93.19 and ID accuracy from 91.23 to 93.53 on the 5% subset.
• With GPT-4o, ReGuide improves All-OoD AUROC from 89.85 to 94.61 and near-OoD (NINCO) AUROC from 93.96 to 96.76 on the 5% subset, though valid response rate drops from 87.58% to 79.57%.
• InternVL2-76B with ReGuide improves All-OoD AUROC from 75.30 to 90.60 on the 5% subset, while FPR@95TPR worsens (44.46 to 64.36).

Measured effects reported in the paper

Surprising findings

• Open-source LVLMs often produce highly biased confidence scores (almost all 0.0 or 100.0), yielding FPR of 100% despite decent accuracy.
• Adding near-ID suggested classes improved ID classification accuracy for InternVL2-26B and GPT-4o rather than hurting it.

Limitations

• Guidelines are insufficient to fully control LVLM behavior and prevent unintended outputs; many response failures remain.
• Image-adaptive class suggestions may be suboptimal depending on image context.
• FPR@95TPR degrades for larger models with ReGuide because ambiguous ID labels get flagged as OoD.

Experimental setup

Datasets: ImageNet200 NINCO SSB-Hard iNaturalist Textures Openimage-O CIFAR10 CIFAR100 MNIST SVHN Places365 Tiny ImageNet

Benchmarks: OpenOOD v1.5 ImageNet200 benchmark (25% and 5% subsets) OpenOOD v1.5 CIFAR10 benchmark (25% subset)

Metrics: AUROC FPR@95TPR FPR@90TPR ID accuracy ECE AURC valid response ratio

Baselines: OpenCLIP SCALE fDBD AugMix+ASH GPT-text (EOE-style) LLaVA-v1.6 GLM-4v QWEN-VL-Chat InternVL2-26B InternVL2-76B Claude 3.5 Sonnet Gemini Pro 1.5

Builds on: CLIP zero-shot OoDD NegLabel EOE OpenOOD v1.5

Closest papers by meaning

• Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization (ICLR 2024 · similarity 0.88)
• Enhanced OoD Detection through Cross-Modal Alignment of Multi-Modal Representations (CVPR 2025 · similarity 0.88)
• CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection (ICML 2024 · similarity 0.88)
• Enhancing Large Vision Language Models with Self-Training on Image Comprehension (NeurIPS 2024 · similarity 0.88)
• AdaNeg: Adaptive Negative Proxy Guided OOD Detection with Vision-Language Models (NeurIPS 2024 · similarity 0.88)
• Reproducible Vision-Language Models Meet Concepts Out of Pre-Training (CVPR 2025 · similarity 0.88)