Research map / Architectures and efficiency / Pruning and efficient tuning
Parameter-Efficient Fine-Tuning of State Space Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks PEFT methods on Mamba and Jamba SSM models and proposes Sparse Dimension Tuning (SDT), which selectively tunes SSM channels and state dimensions and combines with LoRA on linear projections to reach state-of-the-art results.
Problem
Parameter-efficient fine-tuning has been studied mainly for Transformers; it is unclear which PEFT methods work for SSM-based models (Mamba, Jamba) and which parameters of SSM modules should be tuned.
Approach
Benchmarks LoRA variants, prompt/prefix tuning, BitFit and Additional-scan on SSMs, finds LoRA on linear projections is strongest but SSM-module tuning is needed, and proposes SDT: warmup-based selection of trainable channels and state dimensions in the SSM module, combined with LoRA on linear projections, with a theoretical expressivity analysis.
Key results
- On Mamba, LoRA on linear projections reaches 81.2 GLUE average and 61.0 CIFAR-10 accuracy, outperforming prompt tuning, prefix-tuning, BitFit and Additional-scan under <1% trainable parameters.
- On Mamba-1.4B, SDT combined with LoRA improves Spider accuracy to 84.5 versus 83.5 for LoRA and SAMSum R1 51.7 vs 51.6 for LoRA.
- On Jamba-Tiny-319M, SDT with DoRA improves GLUE average to 69.2 versus 67.9 for DoRA alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SDT to S6 while retaining LoRA on linear projections | Mamba · GLUE | Avg.: 80.8 -> 81.1 | Helped |
| Add SDT to S6 while retaining LoRA on linear projections | Mamba · DART | BLEU: 51.0 -> 51.5 | Helped |
| Replace S6 DoRA with SDT, retaining DoRA on linear projections | Mamba · GLUE | Avg.: 80.1 -> 78.2 | Hurt |
| Add SDT to S6 while retaining DoRA on linear projections | Mamba · SAMSum | ROUGE-1: 51.8 -> 52.1 | Helped |
| Replace S6 DoRA with SDT, retaining DoRA on linear projections | Jamba · GLUE | Avg.: 67.9 -> 69.2 | Helped |
| Apply LoRA to linear projections rather than SSM modules | Mamba · GLUE | Avg. Score: 76.9 -> 81.2 | Helped |
| Add LoRA tuning on SSM to LoRA on linear projections | Mamba · GLUE | Avg. Score: 81.2 -> 80.3 | No clear effect |
| Use input-injection prompt tuning instead of linear-projection LoRA | Mamba · GLUE | Avg. Score: 81.2 -> 63.8 | Hurt |
| Use input-injection prefix tuning instead of linear-projection LoRA | Mamba · Spider | accuracy: 57.5 -> 39.7 | Hurt |
| Add SDT to LoRA for dimension selection and parameter updates | Mamba-130M · training | time per epoch (s): 410.0 ± 80.0 -> 330.0 ± 77.5 | Helped |
Surprising findings
- Applying LoRA to SSM modules in addition to linear projections does not improve performance over linear projections alone.
- Prefix-tuning is ineffective on SSM models, as its expressiveness reduces to tuning the initial hidden state.
Limitations
- Theoretical guarantees are restricted to linear activations and require full fine-tuning of the last layer.
- Channel and state selection relies on a warmup stage and parameter magnitude, which may not be optimal.
- Real-world SDT gains are modest and in some cases mixed (e.g., DoRA with SDT lower on GLUE for Mamba).
Experimental setup
Datasets: GLUE DART SAMSum Spider CIFAR-10 CelebA
Benchmarks: GLUE DART SAMSum Spider CIFAR-10 CelebA
Metrics: accuracy Matthews correlation METEOR BLEU ROUGE-1 ROUGE-2 ROUGE-L execution accuracy MSE
Baselines: LoRA DoRA LoRA+ prompt tuning prefix-tuning BitFit Additional-scan full fine-tuning
Builds on: LoRA DoRA LoRA+ prefix-tuning prompt tuning BitFit Mamba Additional-scan
Closest papers by meaning
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.88)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.87)
- Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures (ICLR 2025 · similarity 0.87)
- Theoretical Foundations of Deep Selective State-Space Models (NeurIPS 2024 · similarity 0.87)
- SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models (ICML 2024 · similarity 0.87)
- VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.