Research map / Vision-language and multimodal / Cross-modal learning

MokA: Multimodal Low-Rank Adaptation for MLLMs

NeurIPS 2025 · Vision-language and multimodal › Cross-modal learning · approach: Unimodal-to-multimodal representation transfer · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MokA, a multimodal-aware LoRA variant with modality-specific A matrices, text-to-non-text cross-attention in the low-rank space, and a shared B matrix, evaluated on audio-visual-text, visual-text, and speech-text MLLM tasks.

Problem

Efficient fine-tuning methods for MLLMs borrow LoRA from LLMs with shared parameters across modalities, which underutilizes non-text tokens and ignores modality heterogeneity.

Approach

Split LoRA's down-projection into per-modality matrices (unimodal adaptation), add a cross-attention from non-text tokens to text tokens in the low-rank space (explicit cross-modal adaptation), and keep a single shared up-projection B, so text tokens are left unchanged by the attention.

Key results

• On MUSIC-AVQA with LLaMA2, MokA reaches 75.71 vs 73.41 for LoRA, and on AVE 74.68 vs 69.84.
• On POPE with LLaMA2, MokA reaches 74.23 vs 70.28 for LoRA (a 3.95% gain), at 1.069x forward time.
• On AIR-Bench speech-en with LLaMA2, MokA reaches 39.64 vs 31.75 for LoRA.

Measured effects reported in the paper

Surprising findings

• Partial modality inference with LoRA shows text-only tokens achieve near-full performance while non-text-only inference drops noticeably, suggesting shared LoRA parameters are dominated by text tokens.
• Cross-attention with text as queries (updating text tokens) performs slightly worse than the non-text-as-query design, possibly because modifying text representations harms language modeling.

Limitations

• Gains on Qwen2.5-VL are smaller, attributed to a frozen official visual connector limiting the trainable pathway.
• Extra cross-attention adds forward-time overhead (up to 1.213x in the 4-modality setting).

Experimental setup

Datasets: Video-LLaVA AudioCaps GigaSpeech-M MUSIC-AVQA AVE LLaVA-Instruct-150K A-OKVQA LibriSpeech

Benchmarks: MUSIC-AVQA AVE MME-perception MMBench POPE SEED-Bench MMAU-mini-speech AIR-Bench speech-en MCUB-3

Metrics: accuracy forward time per sample FLOPs memory usage trainable parameter ratio

Baselines: LoRA Multiple LoRA LoRAMoE DoRA HydraLoRA Uni-modal LoRA Uni LoRA + MM LoRA Uni LoRA + MM LoRA + Gate

Builds on: LoRA LoRAMoE DoRA HydraLoRA Linformer

Closest papers by meaning

• Explore How to Inject Beneficial Noise in MLLMs (AAAI 2026 · similarity 0.89)
• Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs (ICLR 2026 · similarity 0.88)
• Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.87)
• MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning (ICLR 2025 · similarity 0.87)
• Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning (ICML 2025 · similarity 0.87)
• CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.87)