AI Paper Atlas

Research map / Architectures and efficiency / Pruning and efficient tuning

Deterministic Differentiable Structured Pruning for Large Language Models

Weiyu Huang, Pengle Zhang, Xiaolu Zhang, JUN ZHOU, Jun Zhu, Jianfei Chen

ICML 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Layer-wise pruning of LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
remove sampling noise from hard-concrete masks and add binarization loss (deterministic HC) (ablation)hard-concrete mask learning · LLaMA-7B 20% sparsitymean zero-shot accuracy: 59.95 -> 61.74Improved
add expanded ReLU mask parameterization (decouple forward gate from sparsity regularizer) (ablation)Det. HC · LLaMA-7B 20% sparsitymean zero-shot accuracy: 61.74 -> 63.92Improved
add knowledge distillation from dense model (ablation)Det. HC + EM · LLaMA-7B 20% sparsitymean zero-shot accuracy: 63.92 -> 64.13Improved
add distillation on DeepSeekMoE-16B (ablation)Det. HC + EM · DeepSeekMoE-16B 20% sparsitymean zero-shot accuracy: 61.80 -> 61.84No clear effect
use layer-wise instead of global sparsity (ablation)DDP · LLaMA-7B 20% sparsitymean accuracy change: -0.48 ppHurt
use expert-wise instead of global sparsity (ablation)DDP · DeepSeekMoE-16B 20% sparsitymean accuracy change: -1.97 ppHurt
train masks on C4 instead of FineWeb-Edu (ablation)DDP · LLaMA-7B 20% sparsitymean accuracy change: -1.14 ppHurt
train masks on LaMini instruction data instead of FineWeb-Edu (ablation)DDP · DeepSeekMoE-16B 20% sparsitymean accuracy change: -0.87 ppHurt
increase mask-learning tokens beyond 60M (ablation)DDP · LLaMA-7B and DeepSeekMoE-16B 20% sparsityperplexity and accuracyNo clear effect
add LoRA recovery tokens to training-free pruning instead of mask optimization (ablation)LoRAP on LLaMA-2-7B · LLaMA-2-7B 20% sparsitymean zero-shot accuracy: 59.44 -> 62.98 at 120M tokens vs DDP 64.82 at 30MMixed
use DDP instead of one-shot pruning with fine-tuningpruned LLMs · DeepSeekMoE-16B 60% sparsitymean accuracy: 51.62 -> 58.18Improved

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 2 of 2 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy