Research map / Architectures and efficiency / Pruning and efficient tuning
MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mamba
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Exploratory study of parameter-efficient fine-tuning for Mamba, adapting Transformer PEFT methods, proposing Mamba-specific variants (Affix-tuning, Partial LoRA, Partial-tuning, Additional-scan), and a two-step search for hybrid PEFT combinations.
Problem
Efficiently adapting pre-trained Mamba (SSM) models to downstream tasks with PEFT remains largely unexplored, and existing PEFT methods designed for Transformers do not directly map onto Mamba's architecture.
Approach
Adapt Transformer PEFT methods to Mamba (ParallelAdapter on out_proj, LoRA on each linear layer), redesign prefix-tuning into architecture-independent Affix-tuning, introduce Partial LoRA over output-feature subsets, Mamba-specific Partial-tuning (bias, A, D, conv1d) with a modified weight decay toward pre-trained weights, and Additional-scan that adds state dimensions to the SSM; combine methods
Key results
- On VTAB-1k with Vim-S, the hybrid PEFT (w/ proj) reaches 72.05% average accuracy, versus 71.52% for the best single method LoRA_p(X) and 68.97% for AdaptFormer on ViT-S.
- On commonsense reasoning, Additional-scan with 0.26% trainable parameters reaches 53.5 average on Mamba 1.4B vs 53.0 for full fine-tuning.
- Affix-tuning (w/o proj) on Mamba 1.4B uses 0.09% of parameters and reaches 53.9 average, the best among the compared methods there.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use two-step hybrid search instead of combining all PEFT methods | Vim-S · VTAB-1k average | accuracy: 67.68 -> 72.05 | Helped |
| Remove embedding projection from Hybrid (ablation) | Vim-S Hybrid · VTAB-1k average | accuracy: 72.05 -> 71.80 | Hurt |
| Apply LoRA only to X output features | Vim-S · VTAB-1k average | accuracy: 71.25 -> 71.52 | Helped |
| Insert affix tokens after in_proj rather than before it (ablation) | Vim-S Affix-tuning · VTAB-1k six-task ablation average | accuracy: 23.85 -> 62.91 | Helped |
| Use three rather than one prefix token after in_proj (ablation) | Vim-S Affix-tuning · VTAB-1k six-task ablation average | accuracy: 62.91 -> 63.51 | Helped |
| Increase affix count from three to four (ablation) | Vim-S Affix-tuning · VTAB-1k six-task ablation average | accuracy: 63.51 -> 56.57 | Hurt |
| Use the proposed neighboring-pretrained-A initialization instead of S4D (ablation) | Vim-S Additional-scan · VTAB-1k six-task ablation average | accuracy: 57.39 -> 61.46 | Helped |
| Place the additional state dimension at the top instead of the bottom with the proposed initialization (ablation) | Vim-S Additional-scan · VTAB-1k six-task ablation average | accuracy: 61.46 -> 61.61 | Helped |
| Use embedding projection for Affix-tuning (ablation) | Mamba · Commonsense reasoning | accuracy: 40.6 -> 43.2; 47.0 -> 47.2; 49.6 -> 50.4 | Mixed |
| Use LoRA_p(X) rather than SLL LoRA | Mamba 130M · Commonsense reasoning average | accuracy: 42.7 -> 43.7 | Helped |
| Use Additional-scan rather than SLL LoRA | Mamba 1.4B · Commonsense reasoning average | accuracy: 52.7 -> 53.5 | Helped |
| Increase the weight-decay strength for Bias-tuning from 1e-4 to 1e-2 (ablation) | Vim-S Bias-tuning · VTAB-1k six-task ablation average | accuracy change: +0.06 -> +0.17 | Helped |
Surprising findings
- Vim keeps improving with more trainable PEFT parameters while ViT PEFT methods overfit early, suggesting PEFT is more effective for Mamba than for Transformers.
- Combining all PEFT methods performs worse than single methods, and high-performing single methods are not necessarily part of the optimal hybrid combination.
Limitations
- Experiments are limited to small models (Vim-S, Mamba up to 1.4B) and the authors leave larger models to future work.
- Vision experiments use VTAB-1k with only 1000 training images per task and ImageNet-1K pre-training rather than ImageNet-21K.
- Code is promised after publication, not released at submission time.
Experimental setup
Datasets: VTAB-1k ImageNet-1K commonsense reasoning tasks (BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, OBQA)
Benchmarks: VTAB-1k commonsense reasoning (8 tasks via Language Model Evaluation Harness)
Metrics: top-1 accuracy average accuracy
Baselines: LoRA AdaptFormer FacT-TK Adapter+ SLL LoRA Full fine-tuning LoSA Head2Toe VQT
Builds on: LoRA Prefix-tuning AdaptFormer BitFit Vision Mamba
Closest papers by meaning
- Parameter Efficient Mamba Tuning via Projector-targeted Diagonal-centric Linear Transformation (CVPR 2025 · similarity 0.88)
- Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiency (NeurIPS 2025 · similarity 0.86)
- Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models (NeurIPS 2024 · similarity 0.86)
- Mixture-of-Control: State-Aware Fine-Tuning for Transformer-based Models (NeurIPS 2026 · similarity 0.86)
- Linear attention is (maybe) all you need (to understand Transformer optimization) (ICLR 2024 · similarity 0.86)
- Mamba Modulation: On the Length Generalization of Mamba Models (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.