Research map / Architectures and efficiency / Pruning and efficient tuning

Re-Imagining Multimodal Instruction Tuning: A Representation View

ICLR 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: Adapter-based parameter-efficient tuning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Multimodal Representation Tuning (MRT), which freezes an LLaVA-style LMM and trains only low-rank representation editors on visual, cross-modality, and textual prefix/suffix representations, achieving strong MME scores with 0.03% trainable parameters and enabling token-level counterfactual control.

Problem

Parameter-efficient multimodal instruction tuning of large multimodal models still lags full fine-tuning, and existing PEFT parameters are abstract and hard to interpret or control.

Approach

Extends intervention-based representation fine-tuning (ReFT) to multimodal models by applying low-rank orthonormal-subspace editors to visual encoder layers, the cross-modality projector, and textual prefix/suffix tokens in the LLM, so that only these editors are trained while the whole model stays frozen.

Key results

• MRT reaches 1580.40 MME with 0.03% trainable parameters, versus 1509.61 for MixLoRA (0.85%) and 1503.98 for M$^2$PT (1.96%), and 1587.26 for full fine-tuning.
• MRT achieves 64.93 MMAvg, above LLaVA full fine-tuning (66.59 is higher, so MRT is below it on MMAvg) and above all PEFT baselines (best baseline M$^2$PT 63.68).
• Counterfactual control via editors reaches 100% counterfact success on CIFAR-10 classes for both misclassification and misalignment while leaving other classes unaffected.

Measured effects reported in the paper

Surprising findings

• Editing the entire textual embedding drops MME sharply to 1233.90 versus 1580.40 for prefix and suffix editing only, suggesting over-editing causes response drift.
• Removing the cross-modality editor has the smallest performance impact among the editor components.

Limitations

• Rank is a manually searched hyperparameter; the search is grid-based and may be time-consuming.
• Performance on SNLI-VE is weaker than several PEFT baselines.
• Controllability relies on fixed prompt templates and generalizes poorly to varied phrasing without a rephraser.

Experimental setup

Datasets: Vision-Flan CIFAR-10 Text-VQA MME VSR CIFAR-100 MNIST SNLI-VE POPE SEED-Bench GQA

Benchmarks: MME Text-VQA VSR SNLI-VE CIFAR-10 CIFAR-100 MNIST POPE SEED-Bench GQA

Metrics: MME score (perception + cognition) accuracy MMAvg controlled counterfact rate inference time training memory and time

Baselines: LoRA APrompt PTUM VPT M$^2$PT MixLoRA ReFT LLaVA_FT

Builds on: ReFT LLaVA LoRA RED

Closest papers by meaning

• Learning to Instruct for Visual Instruction Tuning (NeurIPS 2025 · similarity 0.89)
• Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection (ICLR 2025 · similarity 0.89)
• Context-Aware Multimodal Pretraining (CVPR 2025 · similarity 0.88)
• Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.88)
• Improved Baselines with Visual Instruction Tuning (CVPR 2024 · similarity 0.88)
• MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data (NeurIPS 2026 · similarity 0.88)