Research map / Vision-language and multimodal / Multimodal LLMs
Not All Attention is Needed: Parameter and Computation Efficient Tuning for Multi-modal Large Language Models via Effective Attention Skipping
Qiong Wu, Yiyi Zhou, Weihao Ye, Xiaoshuai Sun, Rongrong Ji
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| skip MHAs instead of FFNs (12 skipped) (ablation) | EAS on LaVIN · ScienceQA (LaVIN-7B tuning) | average accuracy: 82.1 -> 90.0 | Improved |
| restrict skipping candidates to MHA instead of MHA or FFN (12 skipped) (ablation) | EAS on LaVIN · ScienceQA (LaVIN-7B tuning) | average accuracy: 88.6 -> 90.0 | Improved |
| add information exchange (two-path average pooling) to PIA (ablation) | EAS-12 on LaVIN · ScienceQA (LaVIN-7B tuning) | average accuracy: 87.1 -> 90.1 | Improved |
| freeze the average-pooling bias in PIA (ablation) | EAS-12 on LaVIN · ScienceQA (LaVIN-7B tuning) | average accuracy: 90.1 -> 90.0 | No clear effect |
| re-parameterize PIA into FFN at inference (ablation) | EAS-12 on LaVIN · ScienceQA (LaVIN-7B tuning) | inference time (s): 2.97s -> 1.69s | Improved |
| skip 16 MHAs instead of 12 (ablation) | EAS on LaVIN · ScienceQA (LaVIN-7B tuning) | average accuracy: 90.0 -> 88.4 | Hurt |
| skip 8 MHAs instead of 4 (ablation) | EAS on LaVIN · ScienceQA (LaVIN-7B tuning) | average accuracy: 89.4 -> 89.8 | Improved |
| increase PIA hidden dimension from 8 to 64 (ablation) | EAS-8 on LaVIN · ScienceQA (LaVIN-7B tuning) | average accuracy: 87.6 -> 90.1 | Improved |
| replace LaVIN adapters with EAS-12 | LaVIN-7B · ScienceQA (LaVIN-7B tuning) | inference time (s): 3.70s -> 1.69s | Improved |
| use EAS-12 instead of DAS-4 on LaVIN | LaVIN-7B · ScienceQA (LaVIN-7B tuning) | average accuracy: 89.2 -> 90.0 | Improved |
| replace MAM with EAS-8 on LLaVA | LLaVA-7B · ScienceQA (LLaVA-7B tuning) | inference time (s): 2.87s -> 1.98s | Improved |
| skip 12 modules instead of 8 | EAS on LLaVA · ScienceQA (LLaVA-7B tuning) | average accuracy: 88.7 -> 88.1 | Hurt |
Closest papers by meaning
- Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning (ICLR 2024 · similarity 0.90)
- TAIA: Large Language Models are Out-of-Distribution Data Learners (NeurIPS 2024 · similarity 0.89)
- Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.89)
- Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning (ICML 2025 · similarity 0.88)
- A-VL: Adaptive Attention for Large Vision-Language Models (AAAI 2025 · similarity 0.88)
- LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
