Research map / Vision-language and multimodal / Multimodal LLMs
Instruction-Grounded Visual Projectors for Continual Learning of Generative Vision-Language Models
Hyundong Jin, Hyung Jin Chang, Eunwoo Kim
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add adaptive knowledge aggregation (AKA) (ablation) | MVP · Captioning average, Transfer (Vicuna) | RefCLIP-Score: 68.92 -> 76.16 | Improved |
| add adaptive knowledge aggregation (AKA) (ablation) | MVP · Question answering average, Transfer (Vicuna) | accuracy: 44.55 -> 50.89 | Improved |
| add adaptive knowledge aggregation (AKA) (ablation) | MVP · Classification average, Last (Vicuna) | accuracy: 85.53 -> 85.87 | No clear effect |
| add expert pruning (ablation) | MVP w/o AKA · Captioning average, Last (Vicuna) | RefCLIP-Score: 71.55 -> 76.70 | Improved |
| add expert pruning (ablation) | MVP w/o AKA · Question answering average, Last (Vicuna) | accuracy: 75.54 -> 76.17 | Improved |
| add activation bias reduction loss (ablation) | MVP w/o AKA, Prune · Classification average, Last (Vicuna) | accuracy: 3.31 -> 85.92 | Improved |
| add recommendation loss (ablation) | MVP w/o AKA, Prune, L_bias · Classification average, Avg (Vicuna) | accuracy: 58.51 -> 69.59 | Improved |
| add recommendation loss (ablation) | MVP w/o AKA, Prune, L_bias · Question answering average, Avg (Vicuna) | accuracy: 51.39 -> 51.68 | Improved |
| replace single projector with mixture of visual projectors and all components (ablation) | MVP · Classification average, Last (Vicuna) | accuracy: 1.49 -> 85.87 | Improved |
| add L_bias to the method without AKA and pruning (ablation) | MVP w/o AKA, Prune · Classification average, Transfer (Vicuna) | accuracy: 78.06 -> 78.34 | No clear effect |
| use ten GPT-4o generated instructions per task instead of one default instruction | MVP · Classification average, Last (Vicuna) | accuracy: 85.87 -> 84.99 | No clear effect |
| use ten generated instructions per task instead of default | MVP · Captioning average, Transfer (Vicuna) | RefCLIP-Score: 76.16 -> 72.64 | Hurt |
Closest papers by meaning
- Language-Instructed Vision Embeddings for Controllable and Generalizable Perception (ICLR 2026 · similarity 0.89)
- LaViP: Language-Grounded Visual Prompting (AAAI 2024 · similarity 0.89)
- Enhancing Visual Continual Learning with Language-Guided Supervision (CVPR 2024 · similarity 0.88)
- Hawaii: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters (CVPR 2024 · similarity 0.88)
- VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
