Research map / Vision-language and multimodal / Human and embodied modeling
Let ViT Speak: Generative Language-Image Pre-training
Yan Fang, Mengcheng Lan, Zilong Huang, Weixian Lei, Yunqing Zhao, Yujie Zhong, Yingchen Yu, Qi She, Yao Zhao, Yunchao Wei
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add gated attention instead of 4 register tokens (ablation) | GenLIP-So/16 · Attention-sink ablation, GenLIP-So/16, 1B samples | ALL AVG: 55.1 -> 56.3 | Improved |
| add gated attention instead of 1 register token (ablation) | GenLIP-So/16 · Attention-sink ablation, GenLIP-So/16, 1B samples | ALL AVG: 53.3 -> 56.3 | Improved |
| add gated attention to Qwen3-initialized SAIL (ablation) | SAIL · SAIL comparison, Qwen3-0.6B arch, 1B samples | ALL AVG: 53.6 -> 54.8 | Improved |
| train SAIL-g from scratch instead of Qwen3 initialization (ablation) | SAIL-g · SAIL comparison, Qwen3-0.6B arch, 1B samples | ALL AVG: 54.8 -> 56.0 | Improved |
| add diverse-resolution adaptation stage 2 (L/16) (ablation) | GenLIP · Frozen encoder, LLaVA-NeXT Qwen2.5-1.5B | ALL AVG: 55.2 -> 61.5 | Improved |
| add diverse-resolution adaptation stage 2 (So/16) (ablation) | GenLIP · Frozen encoder, LLaVA-NeXT Qwen2.5-1.5B | ALL AVG: 58.9 -> 62.6 | Improved |
| add diverse-resolution adaptation stage 2 (g/16) (ablation) | GenLIP · Frozen encoder, LLaVA-NeXT Qwen2.5-1.5B | ALL AVG: 60.0 -> 65.2 | Improved |
| native-aspect-ratio adaptation stage 2 (ablation) | GenLIP-g/16 · ImageNet-1K frozen attentive probe | top-1 accuracy: 85.2 -> 83.0 | Hurt |
| scale model from L/16 to g/16 (ablation) | GenLIP · Frozen encoder, LLaVA-NeXT Qwen2.5-1.5B | ALL AVG: 61.5 -> 65.2 | Improved |
| GenLIP vs SigLIP at equal 2.0B data | GenLIP-So/16 · Frozen encoder, same 2.0B samples, Qwen2.5-1.5B | ALL AVG: 54.4 -> 57.2 | Improved |
| GenLIP vs OpenVision2 on OCRBench at equal data | GenLIP-So/16 · Frozen encoder, same 2.0B samples, Qwen2.5-1.5B | OCRBench: trails by 6.3 | Hurt |
| GenLIP vs SigLIP2 (40B pairs) with 7B LLM | GenLIP · Frozen encoder, LLaVA-NeXT Qwen2.5-7B | ALL AVG: 69.4 -> 71.8 | Improved |
Closest papers by meaning
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.90)
- Probing Visual Language Priors in VLMs (ICML 2025 · similarity 0.90)
- LMFusion: Adapting Pretrained Language Models for Multimodal Generation (NeurIPS 2025 · similarity 0.89)
- Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.89)
- Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.89)
- VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
