Research map / Large language models / Language, safety and interpretability
VPGTrans: Transfer Visual Prompt Generator across LLMs
Ao Zhang, Hao Fei, Yuan Yao, Wei Ji, Li Li, Zhiyuan Liu, Tat-Seng Chua
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize projector with word-embedding converter before warm-up (ablation) | linear projector warm-up OPT125M->1.3B · COCO caption | CIDEr: 126.1 -> 130.2 | Improved |
| Use 5x learning rate for 1 epoch of projector warm-up (ablation) | projector warm-up · COCO caption | CIDEr: 133.1 vs 132.8 (3 epochs w/o init) | Improved |
| Add projector warm-up before VPG tuning (stage-1) (ablation) | VPG inheritance · VQAv2 | accuracy | Improved |
| Inherit VPG trained on OPT125M instead of training from scratch | OPT350M · COCO caption | convergence speed | Improved |
| Inherit VPG with randomly initialized projector (no warm-up) (ablation) | OPT350M · VQAv2, GQA | accuracy | Hurt |
| Transfer from OPT125M instead of OPT1.3B to OPT2.7B (ablation) | VPGTrans · COCO caption | speed-up rate: 3.3 -> 10.0 | Improved |
| VPGTrans scale-up OPT2.7B->6.7B vs original BLIP-2 OPT6.7B | BLIP-2 ViT-G OPT6.7B · VQAv2 | accuracy: 54.3 -> 57.2 | Improved |
| VPGTrans scale-up OPT2.7B->6.7B vs original BLIP-2 OPT6.7B | BLIP-2 ViT-G OPT6.7B · OKVQA | accuracy: 36.4 -> 39.8 | Improved |
| VPGTrans scale-up OPT2.7B->6.7B vs original BLIP-2 OPT6.7B | BLIP-2 ViT-G OPT6.7B · GPU hours | GPU hours: 631.5 -> 59.0 | Improved |
| VPGTrans FlanT5 XL->XXL vs original BLIP-2 FlanT5XXL | BLIP-2 ViT-G FlanT5XXL · VQAv2 / GQA / OKVQA | accuracy: GQA 44.7 -> 45.0; OKVQA 45.9 -> 45.0 | Mixed |
| VPGTrans FlanT5 XL->XXL cost | BLIP-2 ViT-G FlanT5XXL · GPU hours | GPU hours: 684.0 -> 32.4 | Improved |
| Transfer between two small LLMs (OPT350M<->FlanT5base) | VPGTrans · COCO caption | convergence speed | Hurt |
Closest papers by meaning
- Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions (ICLR 2024 · similarity 0.90)
- VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.85)
- LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptation (ICLR 2025 · similarity 0.85)
- OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging (ICLR 2026 · similarity 0.85)
- Accelerating Pre-training of Multimodal LLMs via Chain-of-Sight (NeurIPS 2024 · similarity 0.85)
- Empowering LLMs to Understand and Generate Complex Vector Graphics (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
