Research map / Vision-language and multimodal / CLIP and zero-shot
Zero-Shot Vision Encoder Grafting via LLM Surrogates
Kaiyu Yue, Vasu Singla, Menglin Jia, John Kirchenbauer, Rifaa Qadri, Zikui Cai, Abhinav Bhatele, Furong Huang, Tom Goldstein
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| preserve early-phase layers (replace late phase 16-26 instead of early phase 1-11) (ablation) | Llama-3B surrogate · POPE binary zero-shot grafting | accuracy: 2.7 -> 80.1 | Improved |
| preserve early-phase layers (replace late phase 16-26 instead of early phase 1-11) (ablation) | Llama-3B surrogate · MMLU (small model) | accuracy: 26.6 -> 58.9 | Improved |
| keep early-phase parameters frozen instead of also tuning alternate early layers (ablation) | Llama-3B surrogate T(16,26) · POPE binary zero-shot grafting | accuracy: 22.3 -> 80.1 | Improved |
| tune extra early layers with translator (T(16,26)*) for the surrogate's own performance (ablation) | Llama-3B surrogate · POPE binary (own encoder) | accuracy: 70.4 -> 84.4 | Improved |
| start layer replacement at the transition point (layer 17) instead of layer 6 (ablation) | Llama-8B surrogate · POPE binary zero-shot grafting | accuracy: 20.6 -> 83.4 | Improved |
| start layer replacement at the transition point (layer 17) instead of layer 12 (ablation) | Llama-8B surrogate · MMBench en zero-shot grafting | accuracy: 45.4 -> 53.5 | Improved |
| zero-shot graft surrogate-trained encoder into Llama-70B instead of using full Llama-70B decoder training | Llama-70B · POPE binary accuracy | accuracy: 84.9 -> 86.8 | Improved |
| zero-shot graft surrogate-trained encoder into Llama-70B instead of full decoder training | Llama-70B · VizWiz | accuracy: 53.0 -> 40.0 | Hurt |
| zero-shot graft surrogate-trained encoder into Llama-70B instead of full decoder training | Llama-70B · MMBench en | accuracy: 71.8 -> 65.6 | Hurt |
| use surrogate-trained encoder as warm start for Llama-70B decoder training (10% data) | Llama-70B · average normalized score | average score: 0.5127 -> 0.6612 | Improved |
| use surrogate-trained encoder with 20% data instead of baseline | Llama-70B · training hours | hours: 34.79 -> 19.17 | Improved |
| train decoder from surrogate-trained encoder vs baseline | Llama-70B · MMLU | accuracy: 79.4 -> 82.5 | Improved |
Closest papers by meaning
- Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance (CVPR 2026 · similarity 0.88)
- FastVLM: Efficient Vision Encoding for Vision Language Models (CVPR 2025 · similarity 0.88)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.88)
- Unveiling Encoder-Free Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.87)
- On the Zero-shot Adversarial Robustness of Vision-Language Models: A Truly Zero-shot and Training-free Approach (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
