Research map / Architectures and efficiency / Pruning and efficient tuning

S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity

NeurIPS 2024 · Architectures and efficiency › Pruning and efficient tuning · approach: Adapter-based parameter-efficient tuning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes S²FT, a structured sparse fine-tuning family that selects attention heads and FFN channels, co-permutes weights to form dense submatrices, and updates only those in place, improving generalization, training efficiency, and multi-adapter serving over LoRA and full FT.

Problem

PEFT methods for LLMs struggle to simultaneously deliver high quality (memorization and generalization), efficient training, and scalable serving; LoRA lags full FT in generalization, while unstructured sparse FT is hard to run efficiently on GPUs.

Approach

Exploit coupled structures (MHA Q/K/V-O heads, FFN up/gate-down channels) to select a sparse subset of heads/channels, co-permute the coupled weight matrices so the selected parts form a dense submatrix, and perform in-place dense gradient updates only on those submatrices ("select sparsely, compute densely").

Key results

• On commonsense reasoning with LLaMA3-8B, S²FT reaches 86.6 average accuracy versus 82.5 for LoRA and 85.2 for DoRA.
• On arithmetic reasoning, S²FT averages 1.3 points above LoRA across base models (e.g., 79.6 vs 77.2 on LLaMA3-8B vs LoRA).
• Compared with full FT, S²FT saves 1.4-3.0x peak memory and speeds up training 1.5-2.7x on LLaMA2-7B/13B on a single A100.

Measured effects reported in the paper

Surprising findings

• Randomly selecting channels is a strong baseline, and selecting large-gradient channels hurts performance, suggesting they hold pre-trained knowledge.
• Among projection matrices, fine-tuning Query/Key contributes far less than Value/Up/Gate or Output/Down.

Limitations

• Experiments limited to LLaMA-family and Mistral models up to 13B; scaling to larger models untested.
• Only language tasks evaluated; not verified for vision or multimodal models.
• No practical serving system implemented; only feasibility shown for serving.

Experimental setup

Datasets: Math10K GSM8K MAWPS AQuA BoolQ PIQA SIQA HellaSwag WinoGrande ARC-easy ARC-challenge OpenbookQA

Benchmarks: BoolQ PIQA SIQA HellaSwag WinoGrande ARC-easy ARC-challenge OpenbookQA MultiArith GSM8K AddSub AQuA

Metrics: accuracy MT-Bench score peak memory training latency adapter switch time parallelism latency

Baselines: Full FT LoRA DoRA Galore LoReFT LISA Prefix-Tuning Series Adapter Parallel Adapter

Builds on: LoRA LISA LoRA+ LoRA-SB structured pruning dependency graphs

Closest papers by meaning

• SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity (ICML 2025 · similarity 0.92)
• Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.90)
• SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs (ICLR 2025 · similarity 0.88)
• WeGeFT: Weight‑Generative Fine‑Tuning for Multi‑Faceted Efficient Adaptation of Large Models (ICML 2025 · similarity 0.88)
• R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.88)
• Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samples (NeurIPS 2025 · similarity 0.87)