Research map / Vision-language and multimodal / Multimodal LLMs
LLaMA-Excitor: General Instruction Tuning via Indirect Feature Interaction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LLaMA-Excitor, a PEFT method that adds bypass Excitor blocks to LLaMA attention to modify attention scores via learnable prompts (indirect feature interaction), preserving pretrained abilities and extending to visual instruction tuning.
Problem
Parameter-efficient instruction tuning of LLMs often injects out-of-distribution features or changes intermediate representations, causing forgetting of inherent abilities when fine-tuning on low-quality instruction data; extending to vision-language also typically requires costly alignment modules.
Approach
Instead of directly modifying hidden states, Excitor blocks add an extra similarity matrix (built from a learnable prompt used as extra Key/Value and a low-rank Query from input tokens) with a cold-start gate to the attention scores, so outputs remain linear combinations of the original Values; the same mechanism takes frozen CLIP visual prompts for multimodal tuning without alignment.
Key results
- On MMLU, LLaMA-Excitor is the only PEFT method that does not drop below LLaMA-7B, reaching +3.12% relative improvement, while full fine-tuning drops -6.63%.
- On COCO Caption, LLaMA-Excitor reaches 49.7 BLEU@4 and 157.5 CIDEr, versus 145.3 CIDEr for BLIP-2 with 129M pretraining samples.
- On ScienceQA, LLaMA-Excitor@336px + LoRA reaches 88.39% average accuracy, versus 85.41% for the 224px variant and about 85.8% for LLaVA w/o pretraining.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune with LLaMA-Excitor on COCO Caption | LLaMA-7B · COCO Caption | CIDEr: +12.2 CIDEr | Helped |
| Fine-tune with LLaMA-Excitor on ScienceQA | LLaMA-7B · ScienceQA test | accuracy: +3.12% performance gain compared with LLaVA | Helped |
| Fine-tune with LLaMA-Excitor instead of LLaVA1.5 | LLaMA2-7B · VQA-v2 | accuracy: +3.6% | Helped |
| Fine-tune with LLaMA-Excitor on Alpaca-52k | LLaMA-7B · MMLU | relative performance change: +3.12% | Helped |
| Fine-tune with full fine-tuning on Alpaca-52k | LLaMA-7B · MMLU | relative performance change: -6.63% | Hurt |
| Combine Excitor with LoRA (ablation) | LLaMA-Excitor · ScienceQA | accuracy: +2.98% | Helped |
| Combine Excitor with LoRA (ablation) | LLaMA-Excitor · MMLU | relative performance change: -4.14% | Hurt |
| Use CLIP-14/L@336px instead of the basic CLIP-16/B image encoder (ablation) | LLaMA-Excitor · ScienceQA | accuracy: +2.76% | Helped |
| Replace the frozen LLM with full fine-tuning (ablation) | LLaMA-Excitor with CLIP-14/L@336px · ScienceQA | accuracy: 85.87 -> 83.20 | Hurt |
| Use linear projection on Query only rather than no projections (ablation) | Text-only Excitor · MMLU | mACC (%): 31.07 -> 35.75 | Helped |
| Add linear projections on both visual Key and Value rather than neither (ablation) | Multimodal Excitor · ScienceQA | mACC (%): 75.96 -> 85.41 | Helped |
| Set low-rank dimension r to 16 rather than r=4 (ablation) | Excitor blocks · MMLU | mACC (%): 29.14 -> 35.75 | Helped |
Surprising findings
- Adding LoRA to Excitor boosts ScienceQA by 2.98% but drops MMLU by 4.14% relative to the original LLaMA-7B.
- Excitor needs no vision-language alignment module and achieves strong captioning results with only COCO training data.
Limitations
- Preliminary study; only evaluated on LLaMA family models, adaptability to other LLMs left for future work.
- Multimodal extension uses only last-layer CLIP features without multi-scale visual prompts.
- Combining with LoRA hurts text-only MMLU performance; GPT-4-judged AlpacaEval uses only 80 questions.
Experimental setup
Datasets: Alpaca 52K COCO Caption ScienceQA LLaVA665k MMLU AlpacaEval VQA-v2 GQA OpenPlatypus Evol-Instruct ARC HellaSwag
Benchmarks: MMLU AlpacaEval COCO Caption ScienceQA VQA-v2 GQA ARC HellaSwag TruthfulQA
Metrics: accuracy relative performance change win rate BLEU@4 CIDEr
Baselines: Full fine-tuning Alpaca-LoRA LLaMA-Adapter ClipCap VL-PET LLaMA-AdapterV2 Qwen-VL-Chat mPLUG-Owl2 BLIP Flamingo BLIP-2 LLaVA
Builds on: LLaMA LLaMA-Adapter LoRA Prefix-tuning zero-init attention
Closest papers by meaning
- Learning to Instruct for Visual Instruction Tuning (NeurIPS 2025 · similarity 0.88)
- Improved Baselines with Visual Instruction Tuning (CVPR 2024 · similarity 0.87)
- Instruct-SkillMix: A Powerful Pipeline for LLM Instruction Tuning (ICLR 2025 · similarity 0.86)
- Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning (ICML 2024 · similarity 0.86)
- ADAPT: Hybrid Prompt Optimization for LLM Feature Visualization (NeurIPS 2026 · similarity 0.86)
- SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.