Research map / Vision-language and multimodal / CLIP and zero-shot
Improving Open Set Recognition via Visual Prompts Distilled from Common-Sense Knowledge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns per-class frame-shaped visual prompts that distill ConceptNet common-sense text (encoded by an LLM) into a frozen visual recognizer to reduce overconfidence in open set recognition.
Problem
Open set recognition suffers from overconfidence, where visual classifiers assign high confidence to unknown-class images because they lack common-sense knowledge.
Approach
Train one frame-shaped visual prompt per known class by contrastively aligning prompted image features with LLM features of ConceptNet common-sense sentences, plus a KL loss pushing predictions toward uniform under mismatched prompts; the base model stays frozen and inference uses a two-stage prompt selection.
Key results
- Baseline VGG32 with visual prompt raises CIFAR10 AUC from 66.7 to 91.2 (+24.5 points).
- ARPL+CS with visual prompt reaches 95.2 AUC on CIFAR10 and 83.1 AUC on TinyImageNet, versus 91.0 and 78.2 without it.
- Ablation on CIFAR10 shows LAlign alone gives 76.0 AUC, and adding LSupp raises it to 89.0 AUC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add visual prompts to ARPL+CS | VGG32 with ARPL+CS · CIFAR10 (C10) | AUC: 91.0 -> 95.2 | Helped |
| Add visual prompts to ARPL+CS | VGG32 with ARPL+CS · CIFAR+10 (C+10) | AUC: 97.1 -> 97.9 | Helped |
| Add visual prompts to ARPL+CS | VGG32 with ARPL+CS · CIFAR+50 (C+50) | AUC: 95.1 -> 97.1 | Helped |
| Add visual prompts to ARPL+CS | VGG32 with ARPL+CS · TinyImageNet (TI) | AUC: 78.2 -> 83.1 | Helped |
| Add visual prompts to ARPL+CS | VGG32 with ARPL+CS · CIFAR10, CIFAR+10, CIFAR+50, TinyImageNet | F1: 75.3 -> 83.3; 82.7 -> 87.5; 75.3 -> 80.1; 67.6 -> 70.2 | Helped |
| Add visual prompts to ARPL+CS | VGG32 with ARPL+CS · CIFAR10, CIFAR+10, CIFAR+50, TinyImageNet | OSCR: 87.9 -> 89.6; 94.7 -> 96.6; 92.9 -> 94.5; 65.9 -> 68.5 | Mixed |
| Add LAlign (ablation) | VGG32 baseline · CIFAR10, CIFAR+10, CIFAR+50, TinyImageNet | AUC: 66.7 -> 76.0; 81.3 -> 87.1; 79.8 -> 85.5; 58.1 -> 67.2 | Helped |
| Add LLS to LAlign (ablation) | VGG32 baseline · CIFAR10, CIFAR+10, CIFAR+50, TinyImageNet | AUC: 76.0 -> 77.5; 87.1 -> 87.6; 85.5 -> 86.0; 67.2 -> 68.0 | Helped |
| Add LSupp to LAlign (ablation) | VGG32 baseline · CIFAR10, CIFAR+10, CIFAR+50, TinyImageNet | AUC: 76.0 -> 89.0; 87.1 -> 93.5; 85.5 -> 94.0; 67.2 -> 75.6 | Helped |
| Add LLS to LAlign + LSupp (ablation) | VGG32 baseline · CIFAR10, CIFAR+10, CIFAR+50, TinyImageNet | AUC: 89.0 -> 91.2; 93.5 -> 94.5; 94.0 -> 94.9; 75.6 -> 77.4 | Helped |
| Use visual prompt size P=10 instead of no visual prompt (ablation) | VGG32 baseline · CIFAR10, CIFAR+10, CIFAR+50, TinyImageNet | AUC: 66.7 -> 80.4; 81.3 -> 88.7; 79.8 -> 87.4; 58.1 -> 66.2 | Helped |
| Increase visual prompt size from P=10 to P=20 (ablation) | VGG32 baseline · CIFAR10, CIFAR+10, CIFAR+50, TinyImageNet | AUC: 80.4 -> 86.3; 88.7 -> 91.9; 87.4 -> 91.2; 66.2 -> 73.4 | Helped |
Limitations
- Overall performance depends on the ConceptNet weight threshold, which must be tuned per dataset.
- Evaluation is limited to small-scale benchmarks (CIFAR and TinyImageNet) with VGG32 backbone.
Experimental setup
Datasets: CIFAR10 CIFAR100 TinyImageNet ConceptNet 5.5
Benchmarks: CIFAR10 CIFAR+10 CIFAR+50 TinyImageNet
Metrics: AUC F1 score OSCR
Baselines: Softmax baseline OpenMax G-OpenMax OSRCI C2AE PROSER OpenHybrid RPL CSSR ARPL+CS DIAS OpenGAN
Builds on: CLIP ARPL DIAS BLIP-2 ConceptNet
Closest papers by meaning
- Just a Few Glances: Open-Set Visual Perception with Image Prompt Paradigm (AAAI 2025 · similarity 0.89)
- From Coarse to Fine-Grained Open-Set Recognition (CVPR 2024 · similarity 0.87)
- Visual In-Context Prompting (CVPR 2024 · similarity 0.86)
- OVMR: Open-Vocabulary Recognition with Multi-Modal References (CVPR 2024 · similarity 0.86)
- LaViP: Language-Grounded Visual Prompting (AAAI 2024 · similarity 0.86)
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.