Research map / Vision-language and multimodal / CLIP and zero-shot
Dynamic Multimodal Prototype Learning in Vision-Language Models
Xingyu Zhu, Shuo Wang, Beier Zhu, Miaoge Li, Yunfan Li, Junfeng Fang, Zhicai Wang, Dongsheng Wang, Hanwang Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Distributed Feature Construction (weighted augmented text and image distributions) to zero-shot CLIP (ablation) | ProtoMM (RN50) · ImageNet | accuracy: 59.81 -> 60.82 | Improved |
| add Multimodal Prototype Learning (visual particles) on top of distributed features (ablation) | ProtoMM (RN50) · ImageNet | accuracy: 60.82 -> 63.76 | Improved |
| add Multimodal Prototype Learning on top of distributed features (ablation) | ProtoMM (ViT-B/16) · ImageNet | accuracy: 68.98 -> 72.01 | Improved |
| add Multimodal Prototype Learning on top of distributed features (ablation) | ProtoMM (ViT-B/16) · Caltech101 | accuracy: 94.65 -> 95.70 | Improved |
| add Multimodal Prototype Learning on top of distributed features (ablation) | ProtoMM (RN50) · Caltech101 | accuracy: 88.16 -> 89.94 | Improved |
| use ProtoMM instead of AWT | CLIP RN50 · Cross-domain benchmark average | accuracy: 62.17 -> 63.10 | Improved |
| use ProtoMM instead of DPE | CLIP ViT-B/16 · Cross-domain benchmark average | accuracy: 69.62 -> 70.70 | Improved |
| use ProtoMM instead of DOTA | CLIP ViT-B/16 · ImageNet OOD average | accuracy: 64.52 -> 65.69 | Improved |
| use ProtoMM instead of DPE | CLIP RN50 · Caltech101 | accuracy: 90.83 -> 89.94 | Hurt |
| use ProtoMM instead of AWT | CLIP RN50 · Pets | accuracy: 88.22 -> 87.92 | Hurt |
| increase confidence threshold tau above 0.8 (ablation) | ProtoMM · ImageNet | accuracy | Hurt |
| increase number of image or text augmentations beyond about 40 or 70 (ablation) | ProtoMM · ImageNet | accuracy | No clear effect |
Closest papers by meaning
- Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models (NeurIPS 2024 · similarity 0.91)
- Toward Modality Gap: Vision Prototype Learning for Weakly-supervised Semantic Segmentation with CLIP (AAAI 2025 · similarity 0.88)
- Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (CVPR 2024 · similarity 0.88)
- VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.88)
- Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editing (NeurIPS 2024 · similarity 0.88)
- Context-Aware Multimodal Pretraining (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
