Research map / Architectures and efficiency / Pruning and efficient tuning
Parameter Efficient Mamba Tuning via Projector-targeted Diagonal-centric Linear Transformation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that Mamba projectors (not the SSM) drive transfer learning and proposes ProDiaL, a PEFT method that indirectly updates pretrained projector weights via a diagonal-centric linear transformation plus low-rank off-diagonal terms.
Problem
Parameter-efficient fine-tuning for Mamba architectures is largely unexplored, and it is unclear which Mamba components matter for downstream adaptation.
Approach
Fine-tuned projector weights can be approximated as W' = W T with T near-identity; ProDiaL freezes W and trains a block-diagonal matrix D, a LoRA-style off-diagonal term, and a scaling factor, targeting only the Mamba projectors.
Key results
- On Mamba-130M, ProDiaL on Out-Proj reaches 44.40 average accuracy across commonsense tasks, versus 43.08 for LoRA on Out-Proj and 42.14 for BitFit.
- On Vim-tiny, ProDiaL on Both-Proj reaches 89.87 average accuracy across StanfordCars, Caltech, and Flowers, above LoRA (89.46) and DoRA (89.29) with similar parameter counts.
- ProDiaL uses under 1% of total parameters while matching or exceeding full fine-tuning of projectors on several tasks (e.g., Vim-small Caltech 97.16 vs 94.94 for Both-Proj FT).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Target projectors rather than the SSM for fine-tuning | Mamba-130M · HellaSwag | accuracy: 40.18 vs 37.52 | Helped |
| Target projectors rather than the SSM for fine-tuning | Vim-tiny · StanfordCars | accuracy: 71.74 vs 70.04 | Helped |
| Add the non-diagonal component ϵ to block-diagonal Db (ablation) | ProDiaL · Caltech | accuracy: 95.24 -> 96.16 | Helped |
| Add the scaling factor s to Db+ϵ (ablation) | ProDiaL · Caltech | accuracy: 96.16 -> 96.24 | Helped |
| Train only diagonal transformations in the SSM instead of fully fine-tuning it (ablation) | Mamba-130M · Caltech | accuracy: 94.01 -> 93.55 | Hurt |
| Train only diagonal transformations in projectors instead of fully fine-tuning them (ablation) | Mamba-130M · Caltech | accuracy: 95.01 -> 95.24 | Helped |
| Use diagonal-only transformations rather than off-diagonal-only transformations (ablation) | Mamba-130M linear transformation T · Caltech | accuracy | Helped |
| Fine-tune projectors rather than embeddings (ablation) | Mamba-130M · Winogrande | accuracy: 40.18 vs 34.39 | Helped |
Surprising findings
- Fine-tuning only the SSM performs worse than fine-tuning projectors, despite the SSM being considered the core of Mamba.
- Fine-tuning SSM and projectors together degrades performance relative to projectors alone in Mamba, unlike Transformers where attention and FFN fine-tuning combine synergistically.
Limitations
- Off-diagonal matrix is still trained via LoRA rather than a dedicated method.
- Evaluated only on Mamba-130M/370M/1.4B and Vim-tiny/small; limited model scale and task coverage.
- Projectors dominate parameter count, which is the premise for focusing on them; generality to other Mamba variants is argued but not exhaustively tested.
Experimental setup
Datasets: PILE ImageNet HellaSwag Winogrande ARC-E ARC-C StanfordCars Caltech Flowers
Benchmarks: HellaSwag Winogrande ARC-E ARC-C StanfordCars Caltech Flowers
Metrics: accuracy
Baselines: Full-FT Linear Probing BitFit Strong LoRA DoRA
Builds on: LoRA DoRA Mamba Vision Mamba (Vim)
Closest papers by meaning
- MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mamba (ICLR 2025 · similarity 0.88)
- Demystify Mamba in Vision: A Linear Attention Perspective (NeurIPS 2024 · similarity 0.87)
- Mamba Modulation: On the Length Generalization of Mamba Models (NeurIPS 2025 · similarity 0.86)
- The Mamba in the Llama: Distilling and Accelerating Hybrid Models (NeurIPS 2024 · similarity 0.85)
- Vision Mamba Mender (NeurIPS 2024 · similarity 0.85)
- Mamba-Adaptor: State Space Model Adaptor for Visual Recognition (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.