Research map / Architectures and efficiency / Pruning and efficient tuning

3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs

NeurIPS 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: Layer-wise pruning of LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes 3BASiL, a 3-block ADMM for one-shot sparse-plus-low-rank decomposition of LLMs with convergence guarantees, plus a transformer-matching (TM) refinement step that improves any sparse or S+LR method.

Problem

One-shot post-training compression of LLMs into sparse plus low-rank form (W ≈ S + LR) suffers large perplexity and accuracy degradation relative to dense models, and existing alternating-minimization methods lack convergence guarantees and handle joint sparse/low-rank optimization poorly.

Approach

Reformulate layer-wise S+LR reconstruction as a 3-block ADMM (sparse S, low-rank L via closed-form rank-r SVD update, auxiliary sparse copy D) with a convergence proof, then refine sparse and low-rank components jointly at the transformer-block level by matching block outputs to the dense block (TM), which is applicable to any decomposition including pure pruning.

Key results

• On Llama-3.2-1B under 3:8+64LR, 3BASiL-TM reaches WikiText2 perplexity 20.75 versus 29.00 for Hf-ALPS (dense 9.75).
• On Meta-Llama-3-8B under 2:4+64LR, 3BASiL-TM reaches WikiText2 perplexity 9.78 versus 11.38 for Hf-ALPS (dense 6.14).
• The paper reports 3BASiL is over 7x faster than Hf-ALPS when compressing Llama3.2-3B on an L40 GPU.

Measured effects reported in the paper

Surprising findings

• Transformer matching applied to pure pruning methods (Wanda, SparseGPT, ALPS) substantially lowers perplexity, e.g., Wanda 2:4 WikiText2 perplexity drops from 26.89 to 11.03.

Limitations

• Quantized or quantized-sparse constraints are not explored.
• Sparsity and rank allocation across layers is not learned; only OWL integration is shown.
• TM on OPT-30B exceeds the 6 A100-hour compression budget.

Experimental setup

Datasets: C4 WikiText2 Penn Treebank PIQA HellaSwag ARC-Easy ARC-Challenge Winogrande RTE OpenbookQA BoolQ

Benchmarks: WikiText2 C4 Penn Treebank PIQA HellaSwag ARC-Easy ARC-Challenge Winogrande RTE OpenbookQA BoolQ

Metrics: perplexity accuracy

Baselines: OATS Hf-SparseGPT Hf-ALPS EoRA Wanda SparseGPT ALPS

Builds on: ADMM HASSLE-free OATS SparseGPT ALPS

Closest papers by meaning

• Compressing Large Language Models using Low Rank and Low Precision Decomposition (NeurIPS 2024 · similarity 0.93)
• SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.92)
• MoDeGPT: Modular Decomposition for Large Language Model Compression (ICLR 2025 · similarity 0.91)
• SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.91)
• Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.90)
• LittleBit: Ultra Low-Bit Quantization via Latent Factorization (NeurIPS 2025 · similarity 0.90)