Research map / Architectures and efficiency / Attention and transformers

CORP: Closed-Form One-shot Representation-Preserving Structured Pruning for Transformers

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Vision transformer pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CORP, a closed-form, one-shot, gradient-free structured pruning method for Transformers that prunes MLP hidden channels and attention Q/K dimensions and compensates via ridge-regression affine/logit reconstruction folded into weights.

Problem

Structured pruning of Transformers usually needs retraining or multi-stage optimization, and naive one-shot structured pruning causes rapid accuracy collapse from uncorrected representation error.

Approach

Treat structured pruning as representation recovery: model removed MLP activations as affine functions of kept ones and approximate pruned Q/K logits with a small matrix M, both solved in closed form by ridge regression on unlabeled calibration data and folded into the remaining projections.

Key results

• On DeiT-Huge at 50% joint MLP+attention sparsity, CORP reaches 83.27% Top-1 (from 84.97 dense) with 42.9% FLOPs reduction and 1.64x throughput.
• On OPT-1.3B at 30% MLP-only pruning, WikiText-2 perplexity goes from 14.60 to 19.36 (attention-only 18.53; joint 25.41).
• Against GRAIL on ViT-Large at ~50% attention pruning, CORP gets 81.98% vs 74.47%; on MLP, 81.80% vs 76.13%.

Surprising findings

• Attention Q/K pruning preserves accuracy better than MLP pruning at matched sparsity in the reported DeiT results, attributed to attention's smaller FLOPs share.
• Accuracy preservation improves strongly with model scale: DeiT-Tiny collapses to 41.36% at 50% joint sparsity while DeiT-Huge retains 83.27%.

Limitations

• Compensation is applied independently per module and ignores cross-module interactions, which accumulate error at high joint sparsity.
• Uniform sparsity ratio across layers ignores layer-wise redundancy differences.
• Performance may degrade under calibration-test distribution shift or very high sparsity; only layer-wise objective, no global optimality guarantee.

Experimental setup

Datasets: ImageNet C4 WikiText-2 NYU Depth V2 ADE20K

Benchmarks: ImageNet WikiText-2 NYU Depth V2 ADE20K

Metrics: Top-1 accuracy FLOPs parameters latency throughput perplexity RMSE delta1 mIoU

Baselines: VBP DC-ViT GRAIL SNOWS Olica KCM

Builds on: Optimal Brain Compression Wanda SparseGPT

Closest papers by meaning

• Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers (CVPR 2024 · similarity 0.90)
• MDP: Multidimensional Vision Model Pruning with Latency Constraint (CVPR 2025 · similarity 0.88)
• TinyFusion: Diffusion Transformers Learned Shallow (CVPR 2025 · similarity 0.86)
• Resource-Efficient Transformer Pruning for Finetuning of Large Models (CVPR 2024 · similarity 0.86)
• Collaborative Multi-Mode Pruning for Vision-Language Models (CVPR 2026 · similarity 0.86)
• The Need for Speed: Pruning Transformers with One Recipe (ICLR 2024 · similarity 0.86)