Research map / Vision-language and multimodal / CLIP and zero-shot
TextRefiner: Internal Visual Feature as Efficient Refiner for Vision-Language Models Prompt Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TextRefiner, a plug-and-play module that builds a local visual cache from CLIP's internal local tokens to refine text prompts of prompt-tuning methods (CoOp, PromptKD) without external LLM knowledge.
Problem
Prompt-learning methods for VLMs learn coarse, class-shared prompts that fail to capture fine-grained class-specific visual concepts, while LLM-based fixes add substantial inference cost.
Approach
A local cache of fixed entries, updated with momentum from the image encoder's local patch tokens, is queried by class text embeddings; the retrieved visual attribute embeddings are fused with the text outputs via a residual linear layer, plus a feature alignment MLP, so no LLM descriptions or extra inference-time text-branch inputs are needed.
Key results
- TextRefiner improves CoOp's 11-benchmark average HM from 71.66% to 76.94% and novel accuracy from 63.22% to 74.32%, surpassing CoCoOp (75.83% HM).
- PromptKD w/ TextRefiner reaches 82.33% average HM, versus 81.27% for LLaMP and 81.09% for PromptKD.
- On the efficiency comparison, PromptKD w/ TextRefiner runs at 12793.26 FPS versus 1473.46 FPS for LLaMP and 20.45 FPS for CoCoOp (single A800).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TextRefiner to PromptKD | PromptKD · 11-benchmark average, base-to-novel | HM: 81.09 -> 82.33 | Helped |
| Add TextRefiner to CoOp | CoOp · 11-benchmark average, base-to-novel | HM: 71.66 -> 76.94 | Helped |
| Add TextRefiner to PromptKD | PromptKD · ImageNet, novel classes | accuracy: 70.96 -> 71.43 | Helped |
| Add TextRefiner to CoOp | CoOp · ImageNet, novel classes | accuracy: 63.22% to 74.32% | Helped |
| Add TextRefiner to CoOp | CoOp · ImageNet-A, cross-domain | accuracy: 49.71 -> 49.77 | No clear effect |
| Add TextRefiner to CoOp | CoOp · ImageNet-Sketch, cross-domain | accuracy: 47.99 -> 48.58 | Helped |
| Add TextRefiner to CoOp (ablation) | CoOp · ImageNet, base-to-novel | HM: 71.92 -> 73.37 | Helped |
| Add semantic loss to TextRefiner (ablation) | CoOp with TextRefiner · ImageNet, base-to-novel | HM: 73.37 -> 73.48 | Helped |
| Add regularization loss to TextRefiner (ablation) | CoOp with TextRefiner · ImageNet, base-to-novel | HM: 73.37 -> 73.02 | Hurt |
| Add regularization loss alongside semantic loss (ablation) | CoOp with TextRefiner · ImageNet, base-to-novel | HM: 73.48 -> 73.56 | Helped |
| Increase the local-cache entry count M (ablation) | TextRefiner · ImageNet, base-to-novel | recognition performance | Mixed |
| Increase the semantic-loss factor λ1 (ablation) | TextRefiner · ImageNet, base-to-novel | HM: 73.10; 73.10; 73.27; 73.56; 73.30 | Mixed |
Surprising findings
- Regularization loss improves base accuracy but reduces novel-class gains from the local cache, as the authors note.
- Performance on novel classes first rises then falls as the fusion coefficient grows, indicating overfitting to seen-class fine-grained information.
Limitations
- Gains on some datasets are small or negative (e.g., CoOp w/ TextRefiner underperforms CoOp on some base-to-novel splits such as StanfordCars novel and FGVCAircraft); performance depends on hyperparameters tuned empirically.
- Novel-class generalization degrades with large fusion coefficient or too many cache entries, indicating overfitting risk in low-data regimes.
- Evaluated only with CLIP ViT-B/16 on the reported benchmarks.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNetV2
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNetV2
Metrics: accuracy harmonic mean FPS
Baselines: CLIP CoOp CoCoOp PromptSRC MaPLe PromptKD LLaMP
Builds on: CoOp CoCoOp PromptKD MaPLe CLIP
Closest papers by meaning
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.90)
- Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.90)
- Compound Text-Guided Prompt Tuning via Image-Adaptive Cues (AAAI 2024 · similarity 0.89)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- PromptKD: Unsupervised Prompt Distillation for Vision-Language Models (CVPR 2024 · similarity 0.89)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.