Research map / Vision-language and multimodal / CLIP and zero-shot
ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models
Chunyuan Li, Haotian Liu, Liunian Harold Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lee, Jianfeng Gao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| initialize linear head from text embeddings (language-init) instead of random, linear probing (ablation) | CLIP ViT-B32 · ELEVATER ICinW (20 datasets) 5-shot | accuracy: 58.09 -> 65.35 | Improved |
| language-init instead of random init, fine-tuning (ablation) | CLIP ViT-B32 · ELEVATER ICinW (20 datasets) 5-shot | accuracy: 29.75 -> 63.29 | Improved |
| language-init, fine-tuning, full-shot (ablation) | CLIP ViT-B32 · ELEVATER ICinW (20 datasets) full-shot | accuracy: 77.77 -> 80.35 | Improved |
| one-projection instead of two-projection language init (ablation) | CLIP ViT-B32 · ELEVATER ICinW (20 datasets) 5-shot | accuracy: 65.35 -> 65.88 | Improved |
| CLIP language-augmented vs supervised ViT, 5-shot LP | CLIP ViT-B32 · ELEVATER ICinW (20 datasets) 5-shot | accuracy: 56.00 -> 65.35 | Improved |
| fine-tuning vs linear probing, 5-shot, language-init (ablation) | CLIP ViT-B32 · ELEVATER ICinW (20 datasets) 5-shot | accuracy: 65.35 -> 63.29 | Hurt |
| fine-tuning vs linear probing, full-shot (ablation) | CLIP ViT-B32 · ELEVATER ICinW (20 datasets) full-shot | accuracy: 78.40 -> 80.35 | Improved |
| fine-tuning vs linear probing for detection (ablation) | GLIP Swin-T · ELEVATER ODinW (35 datasets) 1-shot | AP: 22.2 -> 32.2 | Improved |
| prompt tuning vs linear probing (ablation) | GLIP Swin-T · ELEVATER ODinW (35 datasets) 1-shot | AP: 22.2 -> 29.7 | Improved |
| add external knowledge (WordNet/Wiktionary/GPT3) to class text at zero-shot (ablation) | UniCL/K-LITE · ELEVATER ICinW (20 datasets) zero-shot | accuracy: 27.15 -> 29.92~33.93 | Improved |
| add external knowledge for zero-shot detection (ablation) | GLIP-A · ELEVATER ODinW (35 datasets) zero-shot | AP: 11.53 -> 11.70~13.30 | Improved |
| more GPT3 knowledge items (ablation) | K-LITE · IC vs OD zero-shot | accuracy / AP | Mixed |
Closest papers by meaning
- AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation (NeurIPS 2024 · similarity 0.89)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.89)
- MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly (NeurIPS 2025 · similarity 0.89)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.89)
- Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation (ICML 2026 · similarity 0.89)
- Language Model as Visual Explainer (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
