Research map / Architectures and efficiency / Pruning and efficient tuning

MISA: Memory-Efficient LLMs Optimization with Module-wise Importance Sampling

NeurIPS 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: Low-rank compression and merging · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MISA, a memory-efficient LLM optimizer that partitions each transformer layer into matrix modules and samples them via gradient-norm-based importance sampling with a KL-to-uniform penalty, with an O(1/sqrt(K)) non-convex convergence analysis under Adam.

Problem

Layer-wise (block-coordinate) optimization for memory-efficient LLM fine-tuning and pre-training treats whole transformer layers as uniform blocks and samples them cyclically or uniformly, ignoring module importance and lacking convergence guarantees under Adam with stochastic gradients and multiple

Approach

Decompose each transformer layer into matrix modules (W_q, W_k, W_v, W_o, W_up, W_down, etc.) as optimization blocks, and sample them with softmax probabilities over running averages of scaled squared gradient norms, regularized toward uniform via a KL penalty; only sampled modules keep optimizer states and gradients in memory.

Key results

• On LLaMA3-8B commonsense reasoning, MISA (delta=3%) reaches 86.6 average accuracy vs. 85.9 for LISA and 84.8 for BAdam, with LoRA at 82.5, using 34.4 GB.
• On LLaMA-350M pre-training on C4, MISA (delta=25%) reaches validation perplexity 22.11 versus 21.30 for Adam and 24.34 for GaLore (r=256).
• On Qwen2.5-7B commonsense reasoning, MISA (delta=3%) averages 88.5 vs. 88.2 for LoRA and 88.1 for BAdam.

Measured effects reported in the paper

Surprising findings

• Top-K importance sampling (always picking the most important modules) performs about as poorly as uniform sampling, and Bottom-K sampling hurts accuracy substantially (-4.5 on commonsense).
• Preserving optimizer states across module switches had no significant effect on fine-tuning loss but notably worsened pre-training perplexity.

Limitations

• Pre-training was only done at small scale (130M and 350M); scaling to 7B+ pre-training is unverified.
• Only validated on text-only transformer LLMs; not multimodal or non-transformer architectures.
• Embedding and LM-head layers are not trained in fine-tuning, which is a design choice that departs from LISA.

Experimental setup

Datasets: Commonsense170K (BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC, OBQA) MATH10K Alpaca-GPT4 C4

Benchmarks: BoolQ PIQA SIQA HellaSwag WinoGrande ARC-e ARC-c OBQA GSM8K SVAMP AQuA MAWPS

Metrics: accuracy average accuracy validation perplexity validation loss peak memory (GB) training time

Baselines: LoRA DoRA LISA BAdam GaLore IST Adam ChatGPT

Builds on: BAdam LISA Block coordinate descent Adam GaLore

Closest papers by meaning

• Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining (NeurIPS 2025 · similarity 0.90)
• Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.90)
• Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.90)
• LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning (NeurIPS 2024 · similarity 0.90)
• Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint? (NeurIPS 2025 · similarity 0.90)
• Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.89)