Research map / Vision-language and multimodal / CLIP and zero-shot
FATE: Feature-Adapted Parameter Tuning for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FATE, a PEFT method that projects CLIP image-encoder features into weight-update matrices of the CLIP text encoder, tuning only a small projector for few-shot adaptation.
Problem
Existing parameter-efficient fine-tuning methods for vision-language models learn new adaptation parameters from random initialization, ignoring the informative features the pretrained model already contains.
Approach
Use CLIP vision features (after a two-layer projector) as input-conditioned weight updates W' = alpha*F(F_V) added to frozen text-encoder weight matrices, so visual information is injected into the language encoder's parameter space rather than its prompt space.
Key results
- Base-to-novel: FATE reaches average HM 79.09 across 11 datasets, above MaPLe (78.55) and LASP (78.61) on that average.
- Cross-dataset: FATE gets average 66.95 over 10 target datasets, best among compared methods (MaPLe 66.30).
- Domain generalization: FATE attains 49.58 on ImageNet-A and 77.12 on ImageNet-R, the best among compared methods; ablation shows parameter-space injection beats prompt-space (HM 69.07 vs 64.85 on DTD).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FATE instead of LASP | CLIP ViT-B/16 · Base-to-Novel Generalization (11 datasets, average) | Base accuracy: 82.70 vs 82.11 | No clear effect |
| Use FATE instead of LASP | CLIP ViT-B/16 · Base-to-Novel Generalization (11 datasets, average) | Novel accuracy: 74.90 vs 76.29 | Helped |
| Use FATE instead of LASP | CLIP ViT-B/16 · Base-to-Novel Generalization (11 datasets, average) | HM: 78.61 vs 79.09 | Helped |
| Use FATE instead of MaPLe | CLIP ViT-B/16 · Cross-dataset Evaluation (average) | Accuracy: 66.30 -> 66.95 | Helped |
| Use FATE instead of MaPLe | CLIP ViT-B/16 · ImageNet-Sketch, ImageNet-A, ImageNet-R | Accuracy | Helped |
| Inject visual information into both self-attention and FFN parameter spaces (ablation) | FATE · DTD | HM: 56.37 -> 69.07 | Helped |
| Inject visual information into parameter space rather than prompt space (ablation) | FATE · DTD | HM: 64.85 -> 69.07 | Helped |
| Use 2 FC layers in the projector (ablation) | FATE · DTD | HM: 68.28 -> 69.07 | Helped |
| Use 3 FC layers instead of 2 in the projector (ablation) | FATE · DTD | HM: 69.07 -> 68.88 | Hurt |
| Set scaling factor α to 0.001 (ablation) | FATE · DTD | HM: 65.52 -> 69.07 | Helped |
| Increase scaling factor α from 0.001 to 0.005 (ablation) | FATE · DTD | HM: 69.07 -> 68.65 | Hurt |
| Use concatenation rather than addition for fusion (ablation) | FATE · DTD | HM: 69.07 -> 69.28 | Helped |
Surprising findings
- Injecting visual features into text-encoder weights outperforms injecting them as prompt tokens on DTD (HM 69.07 vs 64.85).
Limitations
- Evaluated only on image classification with ViT-B/16 CLIP
- Ablations reported only on DTD
- Efficiency claims are analytic FLOP estimates rather than measured runtime in the text
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNetV2
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNetV2
Metrics: base accuracy novel accuracy harmonic mean top-1 accuracy
Baselines: CLIP CoOp CoCoOp KgCoOp LASP RPO MaPLe PromptSRC
Builds on: CLIP CoOp CoCoOp MaPLe LoRA
Closest papers by meaning
- Adaptive Parameter Selection for Tuning Vision-Language Models (CVPR 2025 · similarity 0.92)
- VladVA: Discriminative Fine-tuning of LVLMs (CVPR 2025 · similarity 0.89)
- Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages (CVPR 2025 · similarity 0.88)
- ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models (ICLR 2025 · similarity 0.88)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- Spurious Feature Eraser: Stabilizing Test-Time Adaptation for Vision-Language Foundation Model (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.