Research map / Vision-language and multimodal / CLIP and zero-shot

Enhanced OoD Detection through Cross-Modal Alignment of Multi-Modal Representations

CVPR 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: Attribute-based zero-shot learning · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes cross-modal alignment (CMA), a multi-modal fine-tuning regularizer that pulls ID image and text embeddings together on the hypersphere, combined with NegLabel scoring for OoD detection on ImageNet-1k benchmarks.

Problem

Zero-shot and prompt-learning OoD detection with CLIP underuse pretrained knowledge or hurt ID accuracy, and naive multi-modal fine-tuning fails to exploit textual negative concepts, partly because of the modality gap between ID image and text embeddings.

Approach

Adds a CMA loss that globally increases similarity between ID image-text pairs (alignment in the batch softmax denominator), reducing the modality gap; the authors show this corresponds to maximum likelihood of a joint energy-based model on a hypersphere.

Key results

• CMA with NegLabel scoring reaches 19.93% average FPR95 and 95.13% AUROC on the MOS benchmark, versus 27.97% and 93.78% for zero-shot NegLabel.
• CMA achieves 82.64% ImageNet-1k ID accuracy, comparable to FLYP (82.58%) and m2-mix (82.67%), and above ZS MCM/NegLabel (66.60%).
• On OpenOOD v1.5, CMA with NegLabel reaches 26.14%-to-15.29% Far-OoD FPR95 range improvements, with Far-OoD FPR95 of 14.65 for Textures/OpenImage-O subsets and Near-OoD results moderate on SSB-hard.

Measured effects reported in the paper

Surprising findings

• Existing MMFT methods such as FLYP with NegLabel have lower AUROC than zero-shot NegLabel across MOS datasets, showing fine-tuning can degrade use of negative text knowledge.
• On the SSB-hard Near-OoD dataset CMA is only moderate, and FFT with Energy scoring performs best there.

Limitations

• Gains are less pronounced on semantically similar Near-OoD datasets such as SSB-hard.
• Only CLIP ViT-B/16 is evaluated, and the paper plans future work on using auxiliary negative labels during MMFT training.

Experimental setup

Datasets: ImageNet-1k NegMining negative labels

Benchmarks: MOS OpenOOD v1.5 ImageNet-1k

Metrics: FPR95 AUROC ID accuracy

Baselines: MCM NegLabel CLIPN CoOp LoCoOp LP FFT LP-FT FLYP m2-mix

Builds on: CLIP FLYP NegLabel m2-mix MCM

Closest papers by meaning

• WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models (NeurIPS 2024 · similarity 0.90)
• MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities (NeurIPS 2024 · similarity 0.90)
• CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection (ICML 2024 · similarity 0.89)
• $\Delta \mathrm{Energy}$: Optimizing Energy Change During Vision-Language Alignment Improves both OOD Detection and OOD Generalization (NeurIPS 2025 · similarity 0.89)
• DPU: Dynamic Prototype Updating for Multimodal Out-of-Distribution Detection (CVPR 2025 · similarity 0.88)
• AdaNeg: Adaptive Negative Proxy Guided OOD Detection with Vision-Language Models (NeurIPS 2024 · similarity 0.88)