Research map / Vision-language and multimodal / Multimodal LLMs

CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Benchmarks visual fine-tuning (VFT) methods for LLaVA-style MLLMs and proposes CoVFT, which extracts a text-guided context vector inside the ViT and routes visual tokens through dense context-conditioned mixture-of-experts FFNs to mitigate visual preference conflicts.

Problem

Whether to fine-tune or freeze the vision encoder in MLLMs is unresolved; existing VFT methods fail to consistently beat a frozen encoder across multimodal tasks because context-agnostic updates create conflicting visual preferences.

Approach

Introduce a latent context vector c from text-guided cross-attention inside the vision encoder (CVE) and use it to dense-route visual tokens across several expert copies of the ViT FFN (CoMoE), so gradients for different contexts are scaled differently per expert.

Key results

• CoVFT reaches a 61.08% mean over 12 benchmarks on LLaVA-1.5-7B, versus 58.93% for Freeze and 59.29% for full fine-tuning, with gains on all 12 benchmarks over Freeze.
• CoVFT on 7B with mean 61.08% surpasses the Freeze LLaVA-1.5-13B average of 61.43%? No: CoVFT-7B is reported to surpass the 13B average in the abstract, with 13B CoVFT reaching 62.90%.
• Gradient cosine similarity mean rises from 0.076 (full FT) to 0.189 with CoVFT, with standard deviation dropping from 0.112 to 0.051.

Surprising findings

• Existing VFT methods (full fine-tuning, LoRA, BitFit) beat the frozen baseline on only 6, 6, and 9 of 12 benchmarks respectively, indicating instability.
• Dense routing outperforms top-2 sparse routing in CoMoE, which the authors attribute to under-training of experts under sparse routing.

Limitations

• Evaluated only up to 13B parameters and million-scale training data.
• Dense routing over all experts adds parameters and memory compared with sparse MoE.
• Clustering shows some coarse or mixed contexts (e.g., cluster 6), so context modeling may be less discriminative.

Experimental setup

Datasets: LLaVA-665K instruction data LLaVA pre-training 558K caption data Visual Genome

Benchmarks: MME-P MMBench-en MMBench-cn GQA ScienceQA-Image AI2D TextVQA MMVP RealWorldQA CV-Bench COCO CV-Bench ADE CV-Bench Omni3D

Metrics: accuracy mean score across 12 benchmarks

Baselines: Freeze Full fine-tuning VPT SVPT LoRA BitFit QA-ViT Q-MoE

Builds on: LLaVA-1.5 visual prompt tuning LoRA BitFit Mixture-of-Experts

Closest papers by meaning

• Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models (CVPR 2026 · similarity 0.91)
• Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders (ICLR 2026 · similarity 0.91)
• LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.90)
• Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification (ICLR 2025 · similarity 0.90)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.90)
• NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints (NeurIPS 2025 · similarity 0.90)