Research map / Vision-language and multimodal / CLIP and zero-shot
CLIP-driven Coarse-to-fine Semantic Guidance for Fine-grained Open-set Semi-supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CFSG-CLIP, a dual-branch CLIP framework with a semantic filtering module and visual-semantic injection that progressively focuses on fine-grained local cues to separate ID from OOD samples in fine-grained open-set semi-supervised learning.
Problem
Fine-grained open-set semi-supervised learning, where unlabeled data contains fine-grained OOD samples that are visually similar to ID classes, making ID/OOD separation hard; global CLIP features focus on general attributes and miss subtle details.
Approach
A coarse-guidance branch filters patch tokens using coarse text prompts and global visual features; a fine-guidance branch injects these selected local features as replacements for the [CLS] token into the first layers of the CLIP visual encoder, then refines them with fine prompts, with both branches trained via FixMatch-style losses.
Key results
- On FGVCAircraft with 5 labels per class, CFSG-CLIP reaches 61.09% accuracy versus 52.43% for the second-best MaPLe.
- On Stanford Cars with 20 labels per class, CFSG-CLIP reaches 93.08% versus 90.34% for MaPLe.
- On Semi-Aves, CFSG-CLIP reaches 65.63% (ID-only unlabeled) and 63.69% (ID+OOD unlabeled), gains of 5.24 and 4.01 points over the best baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CFSG-CLIP instead of MaPLe | CLIP-based fine-grained OSSL · Stanford Dogs (5 labeled samples/class) | accuracy | No clear effect |
| use CFSG-CLIP instead of MaPLe | CLIP-based fine-grained OSSL · Stanford Dogs (20 labeled samples/class) | accuracy: 1.78% advantage | Helped |
| use CFSG-CLIP instead of MaPLe | CLIP-based fine-grained OSSL · Stanford Cars, CUB-200-2011, and FGVCAircraft | accuracy: 8.66% and 9.23% in 5 and 20 labeled samples | Helped |
| use CFSG-CLIP instead of MaPLe | CLIP-based fine-grained OSSL · Semi-Aves (Uin) | accuracy: 5.24% | Helped |
| use CFSG-CLIP instead of MaPLe | CLIP-based fine-grained OSSL · Semi-Aves (Uin + Uout) | accuracy: 4.01% | Helped |
| add semantic filtering in the coarse-guidance branch (ablation) | CoOp+A · Stanford Cars and FGVCAircraft | accuracy: 86.31 -> 89.15; 50.42 -> 54.65 | Helped |
| add semantic filtering in the fine-guidance branch as well (ablation) | CoOp+A with coarse-branch semantic filtering · FGVCAircraft | accuracy: 54.65 -> 58.01 | Helped |
| add visual-semantic injection (ablation) | CoOp+A with semantic filtering in both branches · Stanford Cars and FGVCAircraft | accuracy: 90.03 -> 90.28; 58.01 -> 61.07 | Helped |
| use global visual feature weighting instead of textual filtering alone (ablation) | Semantic filtering module · Stanford Cars and FGVCAircraft | accuracy: 88.65 -> 89.15; 53.57 -> 54.65 | Helped |
| inject coarse visual-semantic priors in layers 1 to 7 (ablation) | CFSG-CLIP fine-guidance visual encoder · FGVCAircraft | accuracy: 60.95 60.83 | Helped |
| vary top-k patch tokens; use k = 20 (ablation) | CFSG-CLIP semantic filtering module · FGVCAircraft | accuracy: 60.95 61.01 60.77; 61.07; 60.35; 58.85 | Mixed |
| replace the image [CLS] token with local visual features rather than concatenate them (ablation) | CFSG-CLIP visual-semantic injection · Stanford Cars and FGVCAircraft | accuracy: 89.99 -> 90.28; 60.47 -> 61.07 | Helped |
Surprising findings
- Concatenating visual-semantic priors with the [CLS] token performs worse than replacing the [CLS] token with them.
- Injecting priors into deeper encoder layers hurts performance, as the encoder then attends to category-irrelevant details.
Limitations
- Method is designed for fine-grained datasets; applicability to coarse-grained datasets is not validated.
- Potential for rare fine-grained categories is left as future work.
- Ablations are run on only two datasets with a single seed.
Experimental setup
Datasets: CUB-200-2011 Stanford Dogs Stanford Cars FGVCAircraft Semi-Aves
Benchmarks: CUB-200-2011 Stanford Dogs Stanford Cars FGVCAircraft Semi-Aves
Metrics: classification accuracy open-set balanced accuracy
Baselines: CLIP CLIP-LoRA CLIP-Adapter CoOp LoCoOp PLOT MaPLe
Builds on: CLIP CoOp FixMatch IOMatch LoCoOp
Closest papers by meaning
- FG-CLIP: Fine-Grained Visual and Textual Alignment (ICML 2025 · similarity 0.89)
- Anchor-based Robust Finetuning of Vision-Language Models (CVPR 2024 · similarity 0.87)
- FineCLIP: Self-distilled Region-based CLIP for Better Fine-grained Understanding (NeurIPS 2024 · similarity 0.87)
- VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction-Editing Data and Long Captions (NeurIPS 2025 · similarity 0.87)
- Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.87)
- Unknown Prompt the only Lacuna: Unveiling CLIP's Potential for Open Domain Generalization (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.