Research map / Vision-language and multimodal / CLIP and zero-shot
Enhance Vision-Language Alignment with Noise
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PiNI, fine-tuning frozen CLIP by injecting learned positive-incentive noise (variational inference-derived Gaussian noise) into visual and text encoders for few-shot classification.
Problem
Misalignment between visual and linguistic representations when adapting pre-trained CLIP to downstream few-shot classification, with full fine-tuning being costly and prone to overfitting.
Approach
Treats beneficial noise (pi-noise) as a tool: reformulates CLIP inference with prompts as a variable, applies variational inference and Monte Carlo to derive a loss, and learns a Gaussian noise generator (MLP/CNN/cross-attention or learnable embeddings) injected into image features and prompt embeddings of a frozen CLIP.
Key results
- Averaged over 11 datasets, PiNI reaches 80.28% at 16 shots vs 79.53% for CoOp and 78.46% for linear probe.
- On 16-shot ImageNet, PiNI reaches 71.74% vs 71.58% for CoOp and 66.72% for zero-shot CLIP; the largest gain over zero-shot CLIP is 38.18% on EuroSAT.
- In domain generalization (trained on ImageNet), PiNI averages 61.47% across ImageNet-V2/Sketch/A/R, outperforming CoOp (60.61%) and CLIP-Adapter (60.84%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PiNI instead of CoOp | CLIP ViT-B/16 · 11-dataset average, 1-shot | accuracy: 67.46 -> 70.30 | Helped |
| Use PiNI instead of CoOp | CLIP ViT-B/16 · 11-dataset average, 16-shot | accuracy: 79.53 -> 80.28 | Helped |
| Use PiNI instead of Linear Probe | CLIP ViT-B/16 · FGVCAircraft, 16-shot | accuracy: 44.73 -> 43.02 | Hurt |
| Train on ImageNet and evaluate with PiNI rather than CoOp | CLIP ViT-B/16 · ImageNet-A | accuracy: 48.21 -> 48.57 | Helped |
| Inject visual noise into visual features using Cross Attention rather than image-level MLP (ablation) | PiNI · ImageNet, 16-shot | accuracy: 67.43 -> 70.93 | Helped |
| Inject visual noise into visual features using Cross Attention rather than image-level MLP (ablation) | PiNI · Caltech101, 16-shot | accuracy: 94.52 -> 95.82 | Helped |
| Inject visual noise into visual features using Cross Attention rather than image-level MLP (ablation) | PiNI · OxfordPets, 16-shot | accuracy: 90.92 -> 91.99 | Helped |
| Inject visual noise into visual features using Cross Attention rather than image-level MLP (ablation) | PiNI · Food101, 16-shot | accuracy: +0.73 | Helped |
| Use learnable embeddings for noise in prompt embeddings rather than an MLP on text features (ablation) | PiNI · ImageNet, 16-shot | accuracy: 71.14 -> 71.64 | Helped |
| Use learnable embeddings for noise in prompt embeddings rather than an MLP on text features (ablation) | PiNI · Caltech101, 16-shot | accuracy: 95.13 -> 95.61 | Helped |
| Use learnable embeddings for noise in prompt embeddings rather than an MLP on text features (ablation) | PiNI · OxfordPets, 16-shot | accuracy: 91.44 -> 92.69 | Helped |
| Use learnable embeddings for noise in prompt embeddings rather than an MLP on text features (ablation) | PiNI · Food101, 16-shot | accuracy: 85.29 -> 86.98 | Helped |
Surprising findings
- PiNI underperforms linear probe on FGVCAircraft, which the authors attribute to aircraft names rarely seen in CLIP pre-training text.
- Learned noise-perturbed prompt embeddings map to semantically similar words (e.g., photo/photos/pic/picture), which the authors read as richer prompt semantics.
Limitations
- Evaluated only on prompt-based image classification; extension to VQA, detection, generation is future work.
- Underperforms linear probe on fine-grained FGVCAircraft due to pre-training corpus gaps in aircraft names.
- Text noise injection limited by the need to encode image-instanced prompts; authors use shared prompts with learnable or MLP-based generators to avoid this cost.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT ImageNetV2
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT ImageNetV2
Metrics: accuracy
Baselines: Zero-shot CLIP Linear Probe CoOp CLIP-Adapter
Builds on: CLIP CoOp CLIP-Adapter pi-noise
Closest papers by meaning
- Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment (ICLR 2025 · similarity 0.89)
- VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models (ICML 2026 · similarity 0.88)
- Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning (ICML 2024 · similarity 0.88)
- Layer-wise Alignment: Examining Safety Alignment Across Image Encoder Layers in Vision Language Models (ICML 2025 · similarity 0.88)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.88)
- Vision-Language Models are Strong Noisy Label Detectors (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.