Research map / Vision-language and multimodal / CLIP and zero-shot
Context-Aware Multimodal Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LIxP, a context-aware extension of contrastive image-text pretraining (SigLIP/CLIP) using in-batch cross-attention over an image memory buffer, so representations support training-free few-shot adaptation without hurting zero-shot transfer.
Problem
Standard contrastive vision-language pretraining optimizes zero-shot transfer but does not explicitly encourage representations to support few- and many-shot training-free metric-based adaptation at test time.
Approach
Augment the contrastive objective with a cross-attention contextualization step over key/value buffers populated by the current batch's image embeddings (self-attention masked out), trained with a separate loss term and separately learnable temperatures so zero-shot ability is retained while few-shot metric-based adaptation improves.
Key results
- SigLIxP ViT-S/16 (1.5B) gives +5.3% at 32-shot over SigLIP with Tip-Adapter and about 4x sample efficiency (61.1% at 8-shot vs 60.3% at 32-shot), with zero-shot performance within +0.4%.
- On ViT-B/16, SigLIxP with softmax nearest-neighbor training-free classifier reaches 77.9% on ImageNet 16-shot versus 74.7% for the optimization-based DMN.
- Gains hold across ViT-S/16, B/16, L/16 and up to 15B examples, with average 32-shot gains of roughly +2.2% to +3.2% across classifiers while zero-shot changes stay within about 0.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use context-aware SigLIxP pretraining instead of SigLIP | ViT-S/16 with Tip-Adapter · 21 downstream datasets, 32-shot | average accuracy: 60.3% -> 65.6%; +5.3% | Helped |
| Use context-aware pretraining instead of SigLIP | ViT-S/16 with Tip-Adapter · 21 downstream datasets, 8-shot vs 32-shot | accuracy: 61.1% 8-shot vs SigLIP 60.3% 32-shot | Helped |
| Use context-aware SigLIxP pretraining instead of SigLIP | ViT-S/16 zero-shot classifier · zero-shot transfer | accuracy: 46.9% -> 47.3%; +0.4% | No clear effect |
| Remove self-attention masking (ablation) | context-aware pretraining · evaluation subset, 16-shot | accuracy: 64.1 ± 0.5 -> 60.5 ± 0.4 | Hurt |
| Set relative loss weighting α to 0.95 instead of 0.9 (ablation) | context-aware pretraining · evaluation subset, 16-shot | accuracy: 64.1 ± 0.5 -> 62.5 ± 0.3 | Hurt |
| Set relative loss weighting α to 0.8 instead of 0.9 (ablation) | context-aware pretraining · evaluation subset, 16-shot | accuracy: 64.1 ± 0.5 -> 63.8 ± 0.3 | Hurt |
| Set relative loss weighting α to 0.6 instead of 0.9 (ablation) | context-aware pretraining · evaluation subset, 16-shot | accuracy: 64.1 ± 0.5 -> 61.5 ± 0.4 | Hurt |
| Replace separate base and context losses with residual contextualization (ablation) | context-aware pretraining · evaluation subset, 16-shot | accuracy: 64.1 ± 0.5 -> 59.2 ± 0.4 | Hurt |
| Use text representations as context values (ablation) | context-aware pretraining · evaluation subset, 16-shot | accuracy: 64.1 ± 0.5 -> 61.4 ± 0.4 | Hurt |
| Use two successive contextualization steps (ablation) | context-aware pretraining · evaluation subset, 16-shot | accuracy: 64.1 ± 0.5 -> 62.0 ± 0.3 | Hurt |
| Share τ1 and τ2 instead of learning them independently (ablation) | context-aware pretraining · evaluation subset, 16-shot | accuracy: 64.1 ± 0.5 -> 61.8 ± 0.4 | Hurt |
| Share τ2 and τctx instead of learning them independently (ablation) | context-aware pretraining · evaluation subset, 16-shot | accuracy: 64.1 ± 0.5 -> 60.2 ± 0.6 | Hurt |
Surprising findings
- Masking self-attention in the context buffer is essential; without it no context re-use emerges and results match the SigLIP baseline.
- Context-aware post-training on an existing SigLIP checkpoint with +1B examples outperforms a 6B-example SigLIP baseline on 32-shot accuracy.
Limitations
- Gains diminish on datasets already near performance plateaus (e.g. UC Merced at 95.2%).
- Requires a sufficiently large batch size so the in-batch context buffer is informative; smaller active buffers reduce few-shot gains.
Experimental setup
Datasets: WebLI 21 few-shot and domain adaptation benchmarks
Benchmarks: ImageNet-1K ImageNet-Sketch EuroSAT DomainNet CUB 200-2011 DTD Food101 Oxford Pets Stanford Cars SUN397 Places-365 CIFAR100
Metrics: zero-shot accuracy few-shot accuracy (2-32 shots per class) relative gain
Baselines: SigLIP CLIP Tip-Adapter CoOp MaPLe PromptSRC CasPL DMN Linear probe
Builds on: SigLIP CLIP Tip-Adapter Prototypical networks
Closest papers by meaning
- Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning (NeurIPS 2024 · similarity 0.89)
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.89)
- A Practitioner's Guide to Real-World Continual Multimodal Pretraining (NeurIPS 2024 · similarity 0.89)
- CATP: Contextually Adaptive Token Pruning for Efficient and Enhanced Multimodal In-Context Learning (AAAI 2026 · similarity 0.89)
- ProKeR: A Kernel Perspective on Few-Shot Adaptation of Large Vision-Language Models (CVPR 2025 · similarity 0.89)
- No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.