Research map / Architectures and efficiency / Mixture of experts

POP: Online Structural Pruning Enables Efficient Inference of Large Foundation Models

NeurIPS 2026 · Architectures and efficiency › Mixture of experts · approach: Efficient MoE language model systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes POP, a training-free online structural pruning method for FFN channels that uses a prefill-stage tri-state partition (retained/candidate/pruned) and re-selects only candidate channels per decoding step, evaluated on dense LLMs, MoEs, and VLMs.

Problem

Structural pruning of large foundation models fixes its mask at prefilling, so it cannot adapt to context shifts during autoregressive decoding, causing large accuracy drops on generation tasks; full per-step re-evaluation of channel importance is too costly.

Approach

Partition FFN channels after prefilling into retained, candidate, and pruned regions using activation-aware importance quantiles, then at each decoding step recompute importance only for the small candidate set and select within it to meet the target ratio, avoiding global re-evaluation and any calibration or predictor training.

Key results

• At 20% pruning on Llama3.1-8B, POP improves generation accuracy substantially over baselines (average gains of 35.52%, 29.58%, 10.35%, 12.51% over competing methods across models).
• On Llama2-7B at 20% pruning, POP achieves 1.14x end-to-end speedup and 1.38x at 40% pruning with only 2.85% extra FLOPs in MLP.
• On generation tasks at 40% pruning, POP averages 31.90 vs 19.24 for Tyr across Llama2/Llama3.1/Qwen3 models.

Surprising findings

• Offline static pruning methods such as Wanda-sp and FLAP nearly collapse on long-form generation even at moderate ratios.

Limitations

• Pruning applied only to FFN blocks; attention is left unpruned, requiring higher FFN pruning ratios to match parameter reduction.
• Latency at 40% is comparable to Probe Pruning and at 20% attention latency is unchanged, so speedup gains are modest.
• Performance gap to Tyr on some QA settings (about 1.38% average at 20%), which uses offline search.

Experimental setup

Datasets: C4 FineWeb WikiText BoolQ RTE HellaSwag WinoGrande ARC-Easy ARC-Challenge OpenBookQA CoQA MBPP

Benchmarks: BoolQ RTE HellaSwag WinoGrande ARC-Easy ARC-Challenge OpenBookQA CoQA MBPP NQ-Open HumanEval GSM8K

Metrics: accuracy F1 pass@1 exact match perplexity latency FLOPs overhead

Baselines: Wanda-sp FLAP Tyr-the-Pruner Probe Pruning

Builds on: Wanda Probe Pruning FLAP Tyr-the-Pruner

Closest papers by meaning

• Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization (NeurIPS 2025 · similarity 0.92)
• Probe Pruning: Accelerating LLMs through Dynamic Pruning via Model-Probing (ICLR 2025 · similarity 0.91)
• Fluctuation-Based Adaptive Structured Pruning for Large Language Models (AAAI 2024 · similarity 0.89)
• SkipGPT: Each Token is One of a Kind (ICML 2025 · similarity 0.89)
• Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.89)
• The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.89)