Research map / Vision-language and multimodal / Multimodal LLMs

Decoupling Vision and Language: Codebook Anchored Visual Adaptation

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CRAFT, which fine-tunes only the vision encoder of a discrete LVLM against a frozen shared codebook (using a small surrogate LLM for supervision plus test-time token pruning), so the adapted encoder transfers to other LLMs sharing that codebook.

Problem

Vision encoders of LVLMs underperform on domain-specific tasks (medical, fine-grained classification), and adapting them in continuous feature space requires re-aligning the language model each time the encoder changes, and risks forgetting instruction-following.

Approach

Fine-tune only the vision encoder of a discrete LVLM whose quantization uses a frozen shared codebook; a small surrogate LLM sharing that codebook provides end-to-end alignment gradients via straight-through estimation, combined with commitment and contrastive losses and a rarity-weighted test-time token pruning scheme. The adapted encoder can then be plugged into any LLM sharing the codebook with

Key results

• CRAFT with VILA-U-7B surrogate and backbone reaches 68.58% average accuracy vs 55.07% zero-shot VILA-U-7B, and outperforms continuous Vision FT (61.76%) on the ten benchmarks.
• The Qwen2-0.5B-surrogate encoder raises Qwen2.5-3B backbone average accuracy from 46.74% to 59.98%.
• Reported average gain of 13.51% over the original discrete model at best; Qwen2-0.5B surrogate training cuts VRAM by 61.6% and training time by 73.5% versus VILA-7B fine-tuning.

Surprising findings

• Projector and LDIFS fine-tuning and LLM LoRA reduce explanation presence substantially, with LoRA reaching near-zero presence on some datasets.
• Small surrogates can hurt fine-grained classification (e.g., Dogs, Flowers) below zero-shot accuracy.

Limitations

• Gains are inconsistent across datasets; ScienceQA is limited by LLM reasoning, and small surrogates underperform on fine-grained tasks.
• Relies on a shared fixed codebook; the approach is tied to VILA-U-style discrete LVLMs with 16384-entry codebooks, and performance drops sharply with smaller codebooks.
• Judge-based evaluation is limited to up to 350 samples per dataset due to token budget.

Experimental setup

Datasets: IconQA OCRVQA ScienceQA VQARAD EuroSAT Flowers Kvasir PlantVillage Cars Dogs LLaVA CC3M pretrain 595K ShareGPT4V

Benchmarks: IconQA OCRVQA ScienceQA VQARAD EuroSAT Flowers Kvasir PlantVillage Cars Dogs

Metrics: exact-match accuracy correctness explanation presence relevance faithfulness verbosity

Baselines: Vision FT Projector FT LDIFS LLM LoRA VILA-U-7B zero-shot VILA-7B zero-shot

Builds on: VILA-U VQVAE LDIFS LoRA

Closest papers by meaning

• CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.88)
• Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language Models (ICML 2024 · similarity 0.88)
• Unveiling Encoder-Free Vision-Language Models (NeurIPS 2024 · similarity 0.87)
• Data Selection for Fine-tuning Vision Language Models via Cross Modal Alignment Trajectories (ICML 2026 · similarity 0.87)
• VladVA: Discriminative Fine-tuning of LVLMs (CVPR 2025 · similarity 0.87)
• Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation (ICML 2026 · similarity 0.87)