Research map / Multi-agent systems and games / LLM agents
Concept Matching with Agent for Out-of-Distribution Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Concept Matching with Agent (CMA), a training-free zero-shot CLIP-based OOD detector that adds neutral text prompts as "agents" to the ID label set and applies softmax over ID and agent scores to widen the ID/OOD score gap.
Problem
Existing OOD detectors, including CLIP-based ones, rely on a binary ID-vs-OOD relationship and often need training or external data; the paper asks whether neutral prompts acting as agents can improve zero-shot OOD separation.
Approach
Neutral text prompts (agents) are concatenated with ID class-name prompts and included in the softmax denominator while being excluded from the max over ID classes, so OOD images get pulled toward agents and their scores drop, while ID images are barely affected; this forms a vector triangle among ID labels, inputs, and agents.
Key results
- On ImageNet-1k OOD benchmark with CLIP-B/16, CMA averages 32.79% FPR95 and 93.05% AUROC, versus 35.47% / 90.83% for GL-MCM and 42.82% / 90.76% for MCM (zero-shot).
- CMA reaches 93.05 average AUROC, exceeding the best training-based method (LoCoOp, 91.53 AUROC) on the ImageNet-1k benchmark.
- Removing softmax scaling in CMA drops average AUROC from 93.05 to 85.42, showing the agent terms matter through the softmax denominator.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add neutral textual agents to the CLIP-based detector (CMA versus zero-shot baselines) | CLIP-B/16 OOD detection · ImageNet-1k OOD, average | AUROC: 90.83 -> 93.05 | Helped |
| Add neutral textual agents to the CLIP-based detector (CMA versus zero-shot baselines) | CLIP-B/16 OOD detection · ImageNet-1k OOD, average | FPR95: 35.47 -> 32.79 | Helped |
| Add neutral textual agents to the CLIP-based detector | CLIP-B/16 OOD detection · ImageNet-10 ID / ImageNet-20 OOD | AUROC: 98.71 -> 99.19 | Helped |
| Add neutral textual agents to the CLIP-based detector | CLIP-B/16 OOD detection · ImageNet-20 ID / ImageNet-10 OOD | FPR95: 12.91 -> 6.2 | Helped |
| Add neutral textual agents to the CLIP-based detector | CLIP-B/16 OOD detection · ImageNet-10 ID / ImageNet-100 OOD | AUROC: 99.09 -> 99.18 | Helped |
| Add neutral textual agents to the CLIP-based detector | CLIP-B/16 OOD detection · ImageNet-20 ID / ImageNet-100 OOD | FPR95: 13.16 -> 10.32 | Helped |
| Add neutral textual agents to the CLIP-based detector | CLIP-B/16 OOD detection · Waterbirds spurious OOD | AUROC: 98.36 -> 99.01 | Helped |
| Add softmax scaling to CMA (ablation) | CMA with CLIP-B/16 · ImageNet-1k OOD, average | AUROC: 85.42 -> 93.05 | Helped |
| Add softmax scaling to CMA (ablation) | CMA with CLIP-B/16 · ImageNet-1k OOD, average | FPR95: 63.75 -> 32.79 | Helped |
| Increase the CLIP backbone from RN50 to B/16 (ablation) | CMA · ImageNet-1k OOD, average | AUROC: 91.58 -> 93.05 | Helped |
| Increase the CLIP backbone from B/16 to L/14 (ablation) | CMA · ImageNet-1k OOD, average | AUROC: 93.05 -> 93.38 | Helped |
| Vary the temperature parameter (ablation) | CMA with CLIP-B/16 · ImageNet-1k OOD, average | AUROC: 92.58; 92.62; 92.63; 92.62; 92.62; 92.61 | No clear effect |
Surprising findings
- Without softmax scaling, CMA performance collapses (average FPR95 63.75%), so the gain depends on including agents in the denominator.
- The choice of neutral prompt has large effect: best vs worst agent differs by up to 19.95% FPR95 on average.
Limitations
- Neutral prompts semantically close to ID labels can hurt ID detection.
- OOD images very similar to ID labels are hard to separate since external information does not strengthen the model's intrinsic abilities.
- Selecting good neutral prompts is manual and performance varies substantially across agents.
Experimental setup
Datasets: ImageNet-1k iNaturalist SUN Places Textures FashionMNIST STL10 OxfordIIIPet Food101 CUB-200 PlantVillage LFW
Benchmarks: ImageNet-1k OOD benchmark (iNaturalist, SUN, Places, Textures) small-scale ID datasets (12 datasets) near OOD (ImageNet-10/20/100, Waterbirds spurious OOD) ImageNet-1k ID classification accuracy
Metrics: FPR95 AUROC ID ACC
Baselines: MSP Energy ODIN ViM KNN NPOS CoOp LoCoOp MCM GL-MCM
Builds on: MCM CLIP GL-MCM LoCoOp
Closest papers by meaning
- Mixture-of-Agents Enhances Large Language Model Capabilities (ICLR 2025 · similarity 0.85)
- Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaboration (ICLR 2026 · similarity 0.84)
- Chain of Agents: Large Language Models Collaborating on Long-Context Tasks (NeurIPS 2024 · similarity 0.84)
- Improving Knowledge Extraction from LLMs for Task Learning through Agent Analysis (AAAI 2024 · similarity 0.84)
- Offline Training of Language Model Agents with Functions as Learnable Weights (ICML 2024 · similarity 0.84)
- Reflexive Guidance: Improving OoDD in Vision-Language Models via Self-Guided Image-Adaptive Concept Generation (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.