Research map / Vision-language and multimodal / CLIP and zero-shot
ASIF: Coupled Data Turns Unimodal Models to Multimodal without Training
Antonio Norelli, Marco Fumero, Valentino Maiorca, Luca Moschella, Emanuele Rodolà, Francesco Locatello
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use smaller image and text encoders (DeiT-tiny) (ablation) | ASIF · ImageNet zero-shot | accuracy: 60.9 -> 46.5 | Hurt |
| use DeiT-small with smaller sentence encoder (ablation) | ASIF · ImageNet zero-shot | accuracy: 60.9 -> 59.3 | Hurt |
| pretrain image encoder on ImageNet21k (ViT-b16) instead of ImageNet1k (DeiT-base) (ablation) | ASIF · CIFAR100 zero-shot | accuracy: 50.2 -> 63.3 | Improved |
| pretrain image encoder on ImageNet21k (ViT-b16) instead of ImageNet1k (DeiT-base) (ablation) | ASIF · ImageNet zero-shot | accuracy: 60.9 -> 55.4 | Hurt |
| unsupervised DINO visual encoder instead of supervised DeiT (ablation) | ASIF · ImageNet zero-shot | accuracy: 60.9 -> 53.0 | Hurt |
| add 100 EuroSAT image-caption pairs to the multimodal dataset (ablation) | ASIF · EuroSAT | accuracy: 29.4 -> 82.5 | Improved |
| tune k and p on PETS instead of standard k=800, p=8 (ablation) | ASIF · Pets | accuracy: 71.5 -> 72.3 | Improved |
| tune k and p on CIFAR100 instead of standard k=800, p=8 (ablation) | ASIF · CIFAR100 | accuracy: 63.3 -> 64.9 | Improved |
| tune k and p on PETS (ablation) | ASIF · CIFAR100 | accuracy: 63.3 -> 60.9 | Hurt |
| tune k and p on CIFAR100 (ablation) | ASIF · Pets | accuracy: 71.5 -> 63 | Hurt |
| sparsify relative representation to top k entries and exponentiate similarities by p (ablation) | ASIF · zero-shot classification | accuracy | Improved |
| use smaller encoders in scaling experiment up to 1.6M pairs | ASIF · ImageNet zero-shot | accuracy | No clear effect |
Closest papers by meaning
- Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment (CVPR 2025 · similarity 0.90)
- CSA: Data-efficient Mapping of Unimodal Features to Multimodal Features (ICLR 2025 · similarity 0.89)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.89)
- CtrlSynth: Controllable Image Text Synthesis for Data-Efficient Multimodal Learning (ICML 2025 · similarity 0.88)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.88)
- With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
