Research map / Vision-language and multimodal / Human and embodied modeling
Open Ad-hoc Categorization with Contextualized Feature Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OAK, a method that adds a single learnable context token to a frozen CLIP ViT and trains it with GCD clustering plus text-guided pseudo-label regularization, for open ad-hoc categorization across contexts.
Problem
Open ad-hoc categorization: given a context, a few labeled exemplars and unlabeled images, infer novel categories that depend on the context and categorize all images, which common-category methods (CLIP zero-shot, GCD) handle poorly.
Approach
Introduce a per-context learnable token into a frozen CLIP ViT to make attention and features context-aware, trained with GCD's contrastive clustering loss plus a text-guided cross-entropy loss over known and LLM-proposed novel class names using pseudo-labels from SS-KMeans.
Key results
- OAK reaches 87.4% novel accuracy on Stanford Mood, versus 35.4% for CLIP-ZS + LLM vocab and 40.6% for GCD.
- OAK achieves 70.3% Omni accuracy on Stanford, outperforming all baselines (best baseline 52.3%).
- On full-shot Clevr-4, OAK averages 83.3% novel accuracy versus 78.0% for µGCD, with an 11% gain on Texture.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add context-aware attention to the GCD setup (ablation) | OAK · Stanford Action | novel accuracy: 67.8 -> 73.3 | Helped |
| Add context-aware attention to the GCD setup (ablation) | OAK · Stanford Action | known accuracy: 89.4 -> 83.1 | Hurt |
| Add text-guided regularization to the GCD setup (ablation) | OAK · Stanford Action | known accuracy: 89.4 -> 95.7 | Helped |
| Add text-guided regularization to the GCD setup (ablation) | OAK · Stanford Action | novel accuracy: 67.8 -> 63.2 | Hurt |
| Add context-aware attention and text-guided regularization (ablation) | OAK · Stanford Omni | overall accuracy: 52.3 -> 70.3 | Helped |
| Use OAK instead of GCD | Stanford · Stanford Mood | novel accuracy: 40.6 -> 87.4 | Helped |
| Use OAK instead of GCD | Clevr-4 · Clevr-4 Omni | overall accuracy: 22.6 -> 28.5 | Helped |
| Use OAK instead of µGCD | Clevr-4 full-shot · Clevr-4 Texture | novel accuracy: 55.5 -> 66.5 | Helped |
Surprising findings
- CLIP-ZS with LLM-generated vocabulary performs poorly on novel classes (e.g., 0.0% Omni novel-class accuracy in some contexts), showing that class names alone are insufficient.
- Text-guided regularization boosts known accuracy but can reduce novel accuracy by constraining clusters to frozen text embeddings.
Limitations
- Focuses on static image recognition only; video or broader settings not studied.
- Omni accuracy evaluated on only 127 manually annotated overlapping images for Stanford.
- Novel cluster naming is less accurate for unfamiliar contexts such as Mood, Texture and Count.
Experimental setup
Datasets: Stanford (Action, Location, Mood) Clevr-4 (Texture, Color, Shape, Count)
Benchmarks: Stanford Clevr-4
Metrics: known accuracy novel accuracy overall accuracy Omni accuracy
Baselines: CLIP-ZS CLIP-ZS + LLM vocab CLIP-ZS + GT vocab SS-KMeans GCD SimGCD µGCD
Builds on: CLIP generalized category discovery (GCD) visual prompt tuning SimGCD GPC
Closest papers by meaning
- Unknown Prompt the only Lacuna: Unveiling CLIP's Potential for Open Domain Generalization (CVPR 2024 · similarity 0.84)
- OVOD-Agent: A Markov-Bandit Framework for Proactive Visual Reasoning and Self-Evolving Detection (CVPR 2026 · similarity 0.83)
- Cyclic Contrastive Knowledge Transfer for Open-Vocabulary Object Detection (ICLR 2025 · similarity 0.83)
- DoGA: Enhancing Grounded Object Detection via Grouped Pre-Training with Attributes (AAAI 2025 · similarity 0.83)
- Just a Few Glances: Open-Set Visual Perception with Image Prompt Paradigm (AAAI 2025 · similarity 0.83)
- Hierarchical Semantic Tree Anchoring for CLIP-Based Class-Incremental Learning (NeurIPS 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.