Research map / Vision-language and multimodal / Multimodal LLMs

Remodeling Semantic Relationships in Vision-Language Fine-Tuning

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Visual instruction tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LSRM, a cross-attention PEFT framework for vision-language fine-tuning that fuses multilevel CLIP features, adds a learnable diagonal projector, and inherits suppression masks across cross-attention layers, reaching 93.94% on ScienceQA with LLaMA-7B.

Problem

Existing vision-language PEFT methods overlook semantic relationships described in text when aligning vision and language, and final-layer vision encoder features and naive cross-attention lack relational modeling, leading to suboptimal multimodal performance.

Approach

Combines intermediate and final CLIP layer features, a semantic relationship projector with a learnable diagonal scaling after SiLU, and an inheritable cross-attention whose weight matrix M accumulates decay on low-correlation visual-text token pairs across layers, building on MemVP's cross-attention framework.

Key results

• On ScienceQA with LLaMA-7B and CLIP ViT-L/14, LSRM reaches 93.94% average accuracy vs. 93.07% for MemVP (3.9M trainable params).
• With LLaMA-13B, LSRM achieves 94.41% average accuracy vs. 93.78% for MemVP on ScienceQA.
• On COCO captioning with LLaMA-13B, LSRM gets 37.3 BLEU-4 and 123.9 CIDEr vs. 36.6 and 121.6 for MemVP, but below BLIP-2 (43.7 / 145.3).

Surprising findings

• Using only a single intermediate-layer feature (layer 12) combined with the final layer outperformed using more intermediate layers (4 or 8 layers degraded accuracy).
• Delaying activation of inheritable cross-attention until later epochs consistently improved test accuracy.

Limitations

• Gains over MemVP are modest (0.87% on ScienceQA with LLaMA-7B) and single-seed results are not reported.
• Captioning results lag behind full-scale methods such as BLIP-2, and the paper only claims 'comparable' performance.
• Hyperparameters (delta, lambda, shift epoch) are tuned per model and the paper gives limited justification for some choices.

Experimental setup

Datasets: ScienceQA COCO Captions

Benchmarks: ScienceQA COCO Captions

Metrics: accuracy BLEU-4 CIDEr training time inference time

Baselines: MemVP LaVIN LLaVA-LoRA LLaMA-Adapter PILL CoMD MM-CoT UnifiedQA BLIP-2 BLIP VisionLLM-H GPT-4

Builds on: MemVP LLaMA CLIP LaVIN LLaVA-LoRA

Closest papers by meaning

• LoMo: Local Modality Substitution for Deeper Vision-Language Fusion (NeurIPS 2026 · similarity 0.91)
• Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.90)
• CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
• LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.89)
• UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm (CVPR 2026 · similarity 0.89)
• Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.89)