Research map / Vision-language and multimodal / CLIP and zero-shot
Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RLCF, a fully test-time adaptation framework that uses a frozen CLIP model as a reward model and REINFORCE with a mean baseline to adapt VLMs on a single test sample for classification, retrieval, and captioning.
Problem
Zero-shot VLMs suffer from distribution shift at test time; prior test-time adaptation via entropy minimization makes models overconfident in wrong predictions, and captioning/retrieval lack TTA methods.
Approach
Treat the VLM's sampled outputs as actions and use CLIPScore (with a sample-mean baseline so that some behaviors are discouraged) as the reward, optimized with REINFORCE on a single test sample; task-specific sampling (top-K or beam search) and tuned parameters (prompts, image encoder, or projector) make it work across classification, retrieval, and captioning.
Key results
- On ImageNet OOD average, RLCF-S-M with CLIP-ViT-B/16 image encoder tuning reaches 72.64 vs. 57.20 zero-shot and 63.63 for TPT.
- On MS-COCO text-to-image retrieval, RLCF-S raises CLIP-ViT-B/16 R@1 from 33.0 to 38.3.
- On MS-COCO to NoCaps out-domain, RLCF on CLIPCap improves CIDEr from 54.6 to 63.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CLIP-reward RLCF with prompt tuning | CLIP-ViT-B/16 · ImageNet OOD average | top-1 accuracy: 59.28 -> 68.33 | Helped |
| Apply CLIP-reward RLCF with image encoder tuning | CLIP-ViT-B/16 · ImageNet OOD average | top-1 accuracy: 68.72 -> 71.69 | Helped |
| Use an ensemble of CLIP reward models instead of a single CLIP-ViT-L/14 (ablation) | RLCF image encoder tuning · ImageNet OOD average | top-1 accuracy: 71.69 -> 72.45 | Helped |
| Add a momentum buffer to ensemble-reward RLCF (ablation) | RLCF image encoder tuning · ImageNet OOD average | top-1 accuracy: 72.45 -> 72.64 | Helped |
| Use CLIP-reward RLCF | CLIP-ViT-B/16 retrieval · MS-COCO text-to-image retrieval | Recall@1: 33.0 -> 37.3 | Helped |
| Use CLIP-reward RLCF | CLIP-ViT-B/16 retrieval · MS-COCO image-to-text retrieval | Recall@1: 52.5 -> 59.1 | Helped |
| Use an ensemble of CLIP reward models instead of a single CLIP-ViT-L/14 (ablation) | RLCF retrieval · MS-COCO text-to-image retrieval | Recall@1: 37.3 -> 38.3 | Helped |
| Add a momentum buffer to ensemble-reward RLCF (ablation) | RLCF retrieval · Flickr30K image-to-text retrieval | Recall@1: 88.3 -> 88.1 | No clear effect |
| Apply RLCF to CapDec | CapDec · NoCaps in-domain | CIDEr: 62.6 -> 68.0 | Helped |
| Apply RLCF to CLIPCap | CLIPCap · NoCaps out-domain | CIDEr: 54.6 -> 63.8 | Helped |
| Increase classification sampling factor K (ablation) | RLCF image classification · ImageNet-A, ImageNet-V2, and ImageNet-R | top-1 accuracy | No clear effect |
| Increase retrieval sampling factor K (ablation) | RLCF text-image retrieval · MS-COCO and Flickr30K | Recall@1 | Mixed |
Surprising findings
- RLCF with a CLIP-ViT-B/16 student surpasses its CLIP-ViT-L/14 reward model on ImageNet-A/V2/R.
- Knowledge distillation from the CLIP teacher underperforms the feedback approach, which the authors attribute to forcing the student to copy incorrect teacher predictions.
Limitations
- Retrieval and captioning TTA is restricted to updating only the query-side branch or projector to limit cost.
- Performance depends on the quality of CLIP reward models; weaker reward models give smaller gains.
Experimental setup
Datasets: ImageNet ImageNet-A ImageNet-V2 ImageNet-R ImageNet-Sketch MS-COCO Flickr30K NoCaps
Benchmarks: ImageNet ImageNet-A ImageNet-V2 ImageNet-R ImageNet-Sketch MS-COCO Flickr30K NoCaps
Metrics: top-1 accuracy recall@K BLEU@4 CIDEr SPICE RefCLIPScore expected calibration error
Baselines: CoOp CoCoOp TPT TPT+CoOp KD ATKD Pseudo-label MAGIC DeCap
Builds on: test-time prompt tuning (TPT) REINFORCE CLIPScore CapDec CLIPCap MEMO
Closest papers by meaning
- On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.92)
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.92)
- TTRV: Test-Time Reinforcement Learning for Vision Language Models (CVPR 2026 · similarity 0.91)
- Realistic Test-Time Adaptation of Vision-Language Models (CVPR 2025 · similarity 0.91)
- CLIPTTA: Robust Contrastive Vision-Language Test-Time Adaptation (NeurIPS 2025 · similarity 0.91)
- Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.