Research map / Large language models / Instruction and fine-tuning

Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning

ICLR 2024 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that tuning only LayerNorm inside attention blocks (or all LayerNorm alone) of an LLM-based MLLM matches or beats full finetuning and LoRA on five multimodal benchmarks while using far fewer trainable parameters and less GPU memory.

Problem

Adapting LLMs into multimodal LLMs is computationally expensive, and existing parameter-efficient methods such as LoRA or soft prompts often lose performance on multimodal tasks.

Approach

Treat LLM-to-MLLM transformation as domain adaptation and finetune only the LayerNorm weights within each attention block (plus connector, embeddings, and head), with a further LayerNorm-only variant; the authors also link this to lower inter-layer similarity and smaller gradient variance.

Key results

• LayerNorm tuning on a 13B model improves performance by an average of over 20% across five multimodal tasks over LoRA while cutting trainable parameters by 41.9% and GPU memory by 17.6%.
• LayerNorm-simp. (0.004% of parameters) surpasses LoRA by an average of 4.3% across five benchmarks.
• LayerNorm-tuned MLLM with only 20K conversational data outperforms full finetuning on 80K data by 18.4% on two captioning tasks, trailing by 2.5% on MME.

Measured effects reported in the paper

Surprising findings

• Tuning LayerNorm with an ImageNet-pretrained ViT (not language-aligned) still matches or beats full finetuning on captioning tasks.
• Tuning only LayerNorm without the vision-language connector retains most performance, suggesting LayerNorm acts as a domain adaptor.

Limitations

• PEFT methods including LayerNorm tuning are more sensitive to hyperparameters such as learning rate and epochs.
• Results vary notably across LLM variants and benchmarks; e.g., LayerNorm-simp. underperforms on some LLaMA2 settings.
• Gains rely on the specific LLaVA/MiniGPT-4-style architecture and small instruction datasets (80K).

Experimental setup

Datasets: CC3M MiniGPT-4 80K filtered image-text pairs LLaVA-style conversation/detail/reasoning instruction data (20K each) ImageNet (ViT ablation)

Benchmarks: MME VQAv2 MSCOCO captioning Flickr30k captioning POPE

Metrics: MME perception/cognition score VQA accuracy CIDEr POPE accuracy/F1-style yes-no accuracy trainable parameter percentage GPU memory (GB) average layer cosine similarity

Baselines: Full finetuning LoRA Attn. QV Proj. Attn. MLP mPLUG-Owl MiniGPT-4 LLaVA-v0

Builds on: LoRA LLaVA MiniGPT-4 Vicuna CLIP

Closest papers by meaning

• When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.90)
• Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.90)
• LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning (NeurIPS 2024 · similarity 0.90)
• TAIA: Large Language Models are Out-of-Distribution Data Learners (NeurIPS 2024 · similarity 0.89)
• Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.89)
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models (ICLR 2024 · similarity 0.89)