Research map / Label-efficient and robust learning / Recognition and long-tail
Beyond the Seen: Bounded Distribution Estimation for Open-Vocabulary Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes generating unseen-class data with a hierarchy-guided class predictor and caption-based domain generator, then aligning seen/generated distributions via KL and MMD losses on CLIP prompt learning, reporting gains on 11 datasets.
Problem
Open-vocabulary learning estimates the open-environment data distribution using only seen-class data, so the estimation error relative to unseen classes cannot be bounded.
Approach
Generate unseen-class data close to seen-class distribution (via WordNet/LLM-derived hierarchy for class selection and VLM-captioned domain info for style), theoretically bound the distribution estimation error by the generated-to-seen distance, and align output distributions with a KL loss plus MMD loss within a prompt-learning CLIP framework.
Key results
- Base-to-new on EuroSAT: new-class accuracy 87.90 vs 73.90 for PromptSRC (+14.00).
- Average base-to-new harmonic mean 83.36 vs 79.97 for PromptSRC.
- Cross-dataset average 67.68 vs 65.81 for PromptSRC; EuroSAT target gain +9.48.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed method rather than the strongest baseline | CLIP ViT-B/16 · Base-to-base/base-to-new average | base accuracy: 85.19 -> 86.40 | Helped |
| Use the proposed method rather than the strongest baseline | CLIP ViT-B/16 · Base-to-base/base-to-new average | new accuracy: 76.38 -> 80.52 | Helped |
| Use the proposed method rather than the strongest baseline | CLIP ViT-B/16 · Base-to-base/base-to-new average | H: 80.44 -> 83.36 | Helped |
| Use the proposed method rather than the strongest baseline | CLIP ViT-B/16 · Cross-dataset average (ImageNet source) | accuracy: +0.65 | Helped |
| Use the proposed method rather than PromptSRC | CLIP ViT-B/16 · Cross-dataset EuroSAT (ImageNet source) | accuracy: 45.50 -> 54.98 (+9.48) | Helped |
| Add the hierarchy-guided semantic tree to unseen-class prediction (ablation) | Ours · Caltech base-to-new | H: 95.75 -> 97.38 | Helped |
| Select high-similarity rather than lowest-similarity candidate classes (ablation) | Ours · Caltech base-to-new | H: 96.25 -> 97.38 | Helped |
| Use inferred domain information instead of the Picture prompt template (ablation) | Ours · Caltech base-to-new | H: 96.19 -> 97.38 | Helped |
| Use inferred domain information instead of the Photo prompt template (ablation) | Ours · Caltech base-to-new | H: 96.11 -> 97.38 | Helped |
| Use inferred domain information instead of the Image prompt template (ablation) | Ours · Caltech base-to-new | H: 95.87 -> 97.38 | Helped |
| Add distribution alignment (ablation) | Ours · EuroSAT base-to-new | H: 76.10 -> 92.30 | Helped |
| Add distribution alignment (ablation) | Ours · Aircraft base-to-new | H: 29.47 -> 46.37 | Helped |
Surprising findings
- On the source ImageNet dataset the method is slightly below PromptSRC (71.22 vs 71.27) in cross-dataset evaluation.
- Generated-data quality is judged high by human and GPT-4 raters, but generation adds substantial training time (739.2 s vs 101.25 s on EuroSAT).
Limitations
- Generated data carries biases inherited from the LLMs and WordNet used for class selection.
- Cannot handle truly unknown classes (e.g., newly emerging concepts) without external retrieval.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT
Metrics: accuracy (base, new, harmonic mean) distribution distance CLIPScore
Baselines: CLIP CoOp CoCoOp DePT TCP CuTCP DeKg PromptSRC ASPrompt
Builds on: CLIP PromptSRC CoOp CoCoOp Stable Diffusion
Closest papers by meaning
- Unlocking the Power of Open Set: A New Perspective for Open-Set Noisy Label Learning (AAAI 2024 · similarity 0.84)
- CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection (ICML 2024 · similarity 0.83)
- SHiNe: Semantic Hierarchy Nexus for Open-vocabulary Object Detection (CVPR 2024 · similarity 0.83)
- ProxyDet: Synthesizing Proxy Novel Classes via Classwise Mixup for Open-Vocabulary Object Detection (AAAI 2024 · similarity 0.83)
- MiniPLM: Knowledge Distillation for Pre-training Language Models (ICLR 2025 · similarity 0.83)
- By Tying Embeddings You Are Assuming the Distributional Hypothesis (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.