Research map / Label-efficient and robust learning / Recognition and long-tail

Recover and Match: Open-Vocabulary Multi-Label Recognition through Knowledge-Constrained Optimal Transport

CVPR 2025 · Label-efficient and robust learning › Recognition and long-tail · approach: Partial and multi-label learning · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RAM, which recovers local semantics of CLIP with a ladder side adapter (LLA) and matches image regions to labels via a knowledge-constrained entropic optimal transport (KCOT) for open-vocabulary multi-label recognition.

Problem

Open-vocabulary multi-label recognition with CLIP suffers from lost local semantics due to global pre-training, and region-to-label matching relies on naive aggregation (mean pooling, softmax re-weighting), producing spurious predictions from irrelevant regions.

Approach

A parameter-free self-aware attention plus text-guided spatial selection in a memory-efficient ladder side network recovers local features, and region-to-label assignment is cast as an entropic optimal transport problem with label-presence-based unbalanced marginals and a teacher-plan KL term from frozen CLIP.

Key results

• RAM reaches 48.4% F1@3 on NUS-WIDE ZSL and 54.5% F1@3 on MS-COCO ZSL, surpassing the previous best by 1.5 and 1.3 points respectively.
• On PA100K-OV, RAM achieves 66.4% F1@3, a 15.3-point gain over the previous best (DualCoOp 33.5%... ML-Decoder 51.1%).
• Replacing the matching in existing networks, KCOT outperforms average pooling and softmax re-weighting, e.g., 52.2 vs 49.8 ZSL F1@3 on COCO for OT vs re-weighting.

Measured effects reported in the paper

Surprising findings

• Standard one-to-one bipartite matching underperforms one-to-many OT-based matching, indicating that multiple regions per label matter.

Limitations

• Explored only on open-vocabulary multi-label recognition; extension to segmentation left for future work.
• Privacy implications for pedestrian-domain applications noted by authors.

Experimental setup

Datasets: MS-COCO NUS-WIDE RAPv1 PA100K MultiScene MLRSNet

Benchmarks: MS-COCO NUS-WIDE RAP-OV PA100K-OV MultiScene-OV MLRSNet-OV

Metrics: mAP F1@3 F1@5 precision@3 recall@3 precision@5 recall@5

Baselines: DualCoOp DualCoOp++ MKT CoMC QKS BiAM LESA SDLM OVML TGF ML-Decoder RemoteCLIP

Builds on: CLIP CoOp DualCoOp Sinkhorn algorithm LST ladder side tuning

Closest papers by meaning

• TagCLIP: A Local-to-Global Framework to Enhance Open-Vocabulary Multi-Label Classification of CLIP without Training (AAAI 2024 · similarity 0.87)
• Image-to-Image Matching via Foundation Models: A New Perspective for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.86)
• Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.86)
• Towards Open-Vocabulary Semantic Segmentation Without Semantic Labels (NeurIPS 2024 · similarity 0.86)
• Classifier-guided CLIP Distillation for Unsupervised Multi-label Classification (CVPR 2025 · similarity 0.86)
• Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation (NeurIPS 2025 · similarity 0.86)