AI Paper Atlas

Research map / Vision-language and multimodal / CLIP and zero-shot

When Do We Not Need Larger Vision Models?

Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang, Trevor Darrell

ECCV 2024 · Vision-language and multimodal › CLIP and zero-shot · approach: Self-distillation for vision-language pretraining · 55 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
run a small frozen model on multiple image scales (S2) instead of using a bigger modelDINOv2-B vs DINOv2-g · ADE20k semantic segmentationmIoU: 55.5 -> 56.9Improved
apply S2 on base model for ImageNet classification versus larger modelDINOv2-B · ImageNet classificationaccuracy: 85.7 vs 86.3 (DINOv2-L)Hurt
add S2 at 1008 resolution to LLaVA-1.5-7BLLaVA-1.5-7B · V* attribute recognitionaccuracy: 43.5 -> 51.3Improved
add S2 to LLaVA-1.5-7B for math reasoning benchmarkLLaVA-1.5-7B · MathVistaaccuracy: 25.2 -> 25.3No clear effect
scale image scales in MLLM with large vision model instead of model sizeOpenCLIP-L MLLM · MLLM benchmarks (average)accuracyImproved
use multi-scale base model for robotic cube picking instead of base-to-largeMVP · cube picking success ratesuccess rate: about 6% (model size) vs about 20% (image scale)Improved
add S2 pre-training to base model on ImageNet-21k (ViT objective)ViT-B · ImageNet classificationaccuracyImproved
add S2 at pre-training time for DINOv2 objectiveDINOv2-B · ImageNet classificationaccuracyMixed
split large image into regular-size sub-images (ablation)S2-Wrapper · ADE20k semantic segmentationmIoU: 55.7 -> 56.3Improved
concatenate multi-scale features instead of adding them (ablation)S2-Wrapper · ADE20k semantic segmentationmIoU: 55.7 -> 56.9Improved
apply S2 from large model instead of base on OpenCLIP/ViT segmentationS2 scaling · ADE20k semantic segmentationmIoUNo clear effect
add multi-scale features to base model to reconstruct larger model featuresViT-B · ViT-L feature reconstruction (test set)reconstruction ratio: 87.6% -> 100.5%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy