Research map / Vision-language and multimodal / Multimodal LLMs
DIGS: Differentiable, Incremental, Global, Scalable Pruning for Language Models
Bingcong Li, Junlin Xian, TAO JIANG, jwlin jwlin, Cheng ZOU, Geng-Li Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase pruning token budget 5.7B -> 24.5B | DIGS · Qwen2.5VL-3B, 60% sparsity | Avg accuracy: 47.92 -> 51.93 | Improved |
| increase pruning token budget 5.7B -> 24.5B | L0-Dual · Qwen2.5VL-3B, 60% sparsity | Avg accuracy: 44.51 -> 38.97 | Hurt |
| increase pruning token budget 5.7B -> 24.5B | DIGS · Qwen2.5VL-3B, 30% sparsity | Avg accuracy: 56.04 -> 58.75 | Improved |
| increase pruning token budget 5.7B -> 24.5B | L0-Dual · Qwen2.5VL-3B, 30% sparsity | Avg accuracy: 58.26 -> 56.85 | Hurt |
| replace dual ascent with per-step gradient-norm control (DIGS vs L0-Dual) at small budget | L0-Dual · Qwen2.5VL-3B, 5.7B tokens | Avg accuracy: 30%: 58.26 -> 56.04; 60%: 44.51 -> 47.92 | Mixed |
| replace dual ascent with gradient-norm control at 24.5B tokens | L0-Dual · Qwen2.5VL-3B, 60% sparsity | Avg accuracy: 38.97 -> 51.93 | Improved |
| add gradient clipping to L0-Dual (ablation) | L0-Dual · 60% sparsity, ~12B tokens | Avg3: 46.32 -> 45.69 | No clear effect |
| cap lambda at 50 in L0-Dual (ablation) | L0-Dual · 60% sparsity | Avg3: 46.32 -> 52.74 (but ~12B -> ~19B tokens) | Improved |
| use DIGS instead of stabilized L0-Dual (ablation) | L0-Dual+lambda cap · 60% sparsity | Avg3: 52.74 -> 56.29 | Improved |
| lower c_g (larger pruning budget) (ablation) | DIGS · 60% sparsity | Avg3: 45.73 -> 57.37 | Improved |
| add post-pruning distillation (ablation) | DIGS 114B pruned · Qwen2.5VL-3B (33% sparsity) | Avg accuracy: 61.52 -> 63.94 | Improved |
| add FineVision SFT (ablation) | DIGS 114B pruned + distill · Qwen2.5VL-3B (33% sparsity) | Avg accuracy: 63.94 -> 64.32 | Improved |
Closest papers by meaning
- SlimGPT: Layer-wise Structured Pruning for Large Language Models (NeurIPS 2024 · similarity 0.93)
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.92)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.92)
- Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization (NeurIPS 2025 · similarity 0.92)
- SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.92)
- A Simple and Effective Pruning Approach for Large Language Models (ICLR 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
