Research map / Vision-language and multimodal / CLIP and zero-shot
TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment
Bingyi Cao, Koert Chen, Kevis-Kokitsi Maninis, Kaifeng Chen, Arjun Karpur, Ye Xia, Sahil Dua, Tanmaya Dabral, Guangxing Han, Bohyung Han, Joshua Ainslie, Alex Bewley and 7 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace iBOT with iBOT++ (loss on visible tokens too) (ablation) | TIPS ViT-g · ADE150 zero-shot seg | mIoU: 3.5 -> 17.6 | Improved |
| replace iBOT with iBOT++ (ablation) | TIPS ViT-g · PC59 zero-shot seg | mIoU: 14.2 -> 28.6 | Improved |
| replace iBOT with iBOT++ (ablation) | TIPS ViT-g · PASCAL linear-probe seg | mIoU: 82.8 -> 82.5 | No clear effect |
| replace iBOT with iBOT++ (ablation) | TIPS ViT-g · ImageNet KNN | accuracy: 83.2 -> 84.4 | Improved |
| add multi-granularity (Gemini + PaliGemma) captions (ablation) | TIPSv2 ViT-g · Flickr T2I retrieval | Recall@1: 81.7 -> 85.4 | Improved |
| add multi-granularity captions (ablation) | TIPSv2 ViT-g · ADE150 zero-shot seg | mIoU: 17.6 -> 18.1 | Improved |
| add head-only EMA (ablation) | TIPSv2 ViT-g · ADE150 zero-shot seg | mIoU: 18.1 -> 19.1 | Improved |
| add head-only EMA (ablation) | TIPSv2 ViT-g · ImageNet KNN | accuracy: 84.3 -> 84.1 | No clear effect |
| raise iBOT++ masking ratio from 0.0 to 0.75 (ablation) | TIPS ViT-L · ADE150 zero-shot seg | mIoU: 1.0 -> 13.6 | Improved |
| use two CLS tokens with Gemini captions instead of one CLS (web, PaliGemma) (ablation) | TIPSv2 ViT-g · COCO I2T retrieval | Recall@1: 72.3 -> 76.2 | Improved |
| add iBOT++ instead of iBOT (ablation) | CLIP ViT-L · PC60 zero-shot seg | mIoU: 8.0 -> 22.9 | Improved |
| distill ViT-L student from ViT-g teacher | TIPS · ADE150 zero-shot seg | mIoU: 2.6 -> 20.8 | Improved |
Closest papers by meaning
- DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment (CVPR 2025 · similarity 0.91)
- Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.91)
- Text as Partial Constraint: Core–Residual Alignment for Robust Vision–Language Learning (NeurIPS 2026 · similarity 0.90)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.90)
- Improving fine-grained understanding in image-text pre-training (ICML 2024 · similarity 0.89)
- DH-Set: Improving Vision-Language Alignment with Diverse and Hybrid Set-Embeddings Learning (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
