Research map / Architectures and efficiency / Pruning and efficient tuning

OSSCAR: One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimization

ICML 2024 · Architectures and efficiency › Pruning and efficient tuning · approach: Structured network pruning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes OSSCAR, a one-shot structured pruning framework that reformulates a layer-wise reconstruction objective as a combinatorial problem solved by a low-rank-update local search, scaling to OPT models up to 30B for pruning heads and FFN neurons and to CNNs.

Problem

One-shot (post-training, no retraining) structured pruning of large vision and language models, where existing methods are too slow or memory-hungry for billion-parameter models and lose accuracy at high sparsity.

Approach

Formulate structured pruning as a layer-wise quadratic problem with group-structured binary selection, reduce the quadratic matrix to a compact form by exploiting shared coefficients, and solve the combinatorial selection with local swap search that uses low-rank (Woodbury-style) updates to re-evaluate the objective cheaply.

Key results

• On OPT-2.7B at 2x speedup, OSSCAR reaches WikiText perplexity 29.48 versus 3611 for ZipLM (about 125x lower, as stated).
• On ResNet50 on ImageNet at 1.6x speedup, OSSCAR achieves 59.14% accuracy versus 53.15% for the next best method (Lasso).
• OSSCAR prunes OPT-2.7B roughly 6-8x faster than ZipLM (2842 s vs 22839 s total pruning time) and scales to OPT-30B on a single 32GB V100.

Measured effects reported in the paper

Surprising findings

• Magnitude pruning and CHIP collapse to near-chance accuracy on MobileNetV1 at moderate speedups, while Lasso and ThiNet remain much stronger.
• ZipLM required turning off its SPDY search to run on OPT models, since the full search was estimated at around 200 days for OPT-2.7B.

Limitations

• Experiments are one-shot only; no fine-tuning results are reported, so gains over retraining-based methods are not measured.
• ZipLM was modified (SPDY search disabled, CPU offloading added) for OPT models, so the baseline may not reflect its original performance.
• Pruning is restricted to attention heads and FFN intermediate neurons for LLMs, and to channels for CNNs; the paper reports only one-shot results with 128 calibration segments in the main LLM table.

Experimental setup

Datasets: C4 CIFAR10 ImageNet

Benchmarks: CIFAR10 ImageNet WikiText2 PTB C4 validation

Metrics: accuracy perplexity inference speedup pruning time

Baselines: Magnitude pruning MP+ ZipLM CHIP FPGM Lasso ThiNet

Builds on: ZipLM Optimal Brain Surgeon ThiNet CHIP

Closest papers by meaning

• Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.91)
• Preserving Deep Representations in One-Shot Pruning: A Hessian-Free Second-Order Optimization Framework (ICLR 2025 · similarity 0.89)
• ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models (ICLR 2024 · similarity 0.89)
• ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.89)
• Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization (NeurIPS 2025 · similarity 0.89)
• EfficientLLaVA: Generalizable Auto-Pruning for Large Vision-language Models (CVPR 2025 · similarity 0.88)