Research map / Vision-language and multimodal / CLIP and zero-shot
When Do We Not Need Larger Vision Models?
Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang, Trevor Darrell
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| run a small frozen model on multiple image scales (S2) instead of using a bigger model | DINOv2-B vs DINOv2-g · ADE20k semantic segmentation | mIoU: 55.5 -> 56.9 | Improved |
| apply S2 on base model for ImageNet classification versus larger model | DINOv2-B · ImageNet classification | accuracy: 85.7 vs 86.3 (DINOv2-L) | Hurt |
| add S2 at 1008 resolution to LLaVA-1.5-7B | LLaVA-1.5-7B · V* attribute recognition | accuracy: 43.5 -> 51.3 | Improved |
| add S2 to LLaVA-1.5-7B for math reasoning benchmark | LLaVA-1.5-7B · MathVista | accuracy: 25.2 -> 25.3 | No clear effect |
| scale image scales in MLLM with large vision model instead of model size | OpenCLIP-L MLLM · MLLM benchmarks (average) | accuracy | Improved |
| use multi-scale base model for robotic cube picking instead of base-to-large | MVP · cube picking success rate | success rate: about 6% (model size) vs about 20% (image scale) | Improved |
| add S2 pre-training to base model on ImageNet-21k (ViT objective) | ViT-B · ImageNet classification | accuracy | Improved |
| add S2 at pre-training time for DINOv2 objective | DINOv2-B · ImageNet classification | accuracy | Mixed |
| split large image into regular-size sub-images (ablation) | S2-Wrapper · ADE20k semantic segmentation | mIoU: 55.7 -> 56.3 | Improved |
| concatenate multi-scale features instead of adding them (ablation) | S2-Wrapper · ADE20k semantic segmentation | mIoU: 55.7 -> 56.9 | Improved |
| apply S2 from large model instead of base on OpenCLIP/ViT segmentation | S2 scaling · ADE20k semantic segmentation | mIoU | No clear effect |
| add multi-scale features to base model to reconstruct larger model features | ViT-B · ViT-L feature reconstruction (test set) | reconstruction ratio: 87.6% -> 100.5% | Improved |
Closest papers by meaning
- Scalable Pre-training of Large Autoregressive Image Models (ICML 2024 · similarity 0.87)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.87)
- Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning (ICML 2024 · similarity 0.87)
- Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model (NeurIPS 2024 · similarity 0.87)
- LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token (ICLR 2025 · similarity 0.86)
- Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
