Research map / Vision-language and multimodal / CLIP and zero-shot
Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dual Memory Networks (DMN), a dynamic memory of historical test features plus a static memory of labeled training features, that unifies zero-shot, few-shot, and training-free few-shot CLIP adaptation without external training data.
Problem
Existing CLIP adaptation methods are tailored to only one or two of three settings (zero-shot, few-shot, training-free few-shot), and zero-shot methods typically rely on external synthetic or retrieved data.
Approach
Two cache-based memory networks with a shared cross-attention readout: a dynamic memory stores pseudo-labeled test features online during testing, and a static memory caches training features; both use optional zero-initialized residual projection layers so the method works training-free or with light training.
Key results
- In zero-shot setting, DMN-ZS reaches 63.71% mean accuracy with ResNet-50 and 70.72% with ViT-B/16 over 11 datasets, outperforming the best non-external-data competitor (CALIP, TPT) by 3.40% and 5.27%.
- DMN-ZS (63.71 mean with RN50) surpasses CaFo, which uses external synthetic training data, by 1.48%.
- On ImageNet-A with ViT-B/16, DMN-ZS reaches 58.28% vs 54.77% for TPT, showing robustness to natural distribution shifts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use dynamic and static memory networks together (ablation) | DMN-TF · training-free few-shot classification | accuracy | Helped |
| Add dynamic memory of historical test samples (ablation) | DMN-TF · training-free few-shot classification | accuracy | Helped |
| Add static memory of labeled training samples (ablation) | DMN-TF · training-free few-shot classification | accuracy | Helped |
| Increase dynamic memory length (ablation) | DMN · classification benchmarks | accuracy | Helped |
| Add Q, K, V, and O projection layers (ablation) | DMN · classification benchmarks | accuracy | Helped |
| Add output projection layer O (ablation) | DMN · classification benchmarks | accuracy | Helped |
| Vary β in the memory readout function (ablation) | DMN · classification benchmarks | accuracy | Mixed |
| Use DMN-ZS instead of the strongest listed zero-shot baseline | CLIP ResNet50 · 11-dataset zero-shot mean | accuracy: 63.71 vs. 62.23 | Helped |
| Use DMN-ZS instead of the strongest listed zero-shot baseline | CLIP ViT-B/16 · 11-dataset zero-shot mean | accuracy: 70.72 vs. 65.91 | Helped |
| Use DMN-ZS on ImageNet-A | CLIP ResNet50 · ImageNet-A | accuracy: 28.57 vs. 31.06 | Hurt |
| Use DMN-ZS on ImageNet-V2 | CLIP ResNet50 · ImageNet-V2 | accuracy: 56.12 vs. 55.80 | Helped |
Surprising findings
- Dynamic memory of unlabeled historical test features alone outperforms static memory of labeled training features in the training-free few-shot setting.
- Simply concatenating training features into the dynamic memory degrades performance as the dynamic memory fills, motivating a separate static memory.
Limitations
- Two external memories consume storage (204.8MB dynamic and 65.5MB static for 16-shot ImageNet).
- Few-shot DMN does not uniformly beat competitors on every individual dataset.
- Dynamic memory relies on pseudo-labels from the text classifier, whose errors can propagate into the cache.
Experimental setup
Datasets: ImageNet Flowers102 DTD OxfordPets StanfordCars UCF101 Caltech101 Food101 SUN397 FGVC-Aircraft EuroSAT ImageNet-V2
Benchmarks: ImageNet Flowers102 DTD OxfordPets StanfordCars UCF101 Caltech101 Food101 SUN397 FGVC-Aircraft EuroSAT ImageNet-V2
Metrics: classification accuracy GFLOPs learnable parameters inference latency
Baselines: CLIP TPT CALIP DiffTPT CuPL SuS-X CaFo Tip-Adapter Tip-X APE CoOp CoCoOp
Builds on: CLIP Tip-Adapter TPT CALIP
Closest papers by meaning
- Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages (CVPR 2025 · similarity 0.91)
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.90)
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.89)
- Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models (ICLR 2024 · similarity 0.89)
- ProKeR: A Kernel Perspective on Few-Shot Adaptation of Large Vision-Language Models (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.