Research map / Architectures and efficiency / Pruning and efficient tuning
Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers
Sotiris Anagnostidis, Dario Pavllo, Luca Biggio, Lorenzo Noci, Aurelien Lucchi, Thomas Hofmann
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add learned dynamic context pruning (~80% sparsity) | GPT-2-small · Wikipedia+BookCorpus context 1000 | perplexity: -0.085 | No clear effect |
| add dynamic context pruning instead of local or sparse attention at same sparsity | GPT-2-small · Wikipedia+BookCorpus | perplexity | Improved |
| add dynamic pruning to GPT-2-small for long context throughput | GPT-2-small · long context | throughput: +98% throughput for 0.316 perplexity loss | Improved |
| add dynamic pruning to GPT-2-medium | GPT-2-medium · context 1000 | throughput: +189% throughput for 0.084 perplexity loss | Improved |
| train only interaction parameters, freeze rest of network (ablation) | dynamic context pruning · Wikipedia+BookCorpus | perplexity: +9.285 perplexity | Hurt |
| tie dropping decisions across depth (ablation) | dynamic context pruning · GPT-2 models | perplexity | Hurt |
| use regular sigmoid with temperature instead of alpha-sigmoid (ablation) | dynamic context pruning · GPT-2-small | training stability | Hurt |
| increase interaction dimension r (ablation) | dynamic context pruning · GPT-2-small | sparsity at fixed perplexity | Improved |
Closest papers by meaning
- Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows (NeurIPS 2025 · similarity 0.90)
- Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers (NeurIPS 2025 · similarity 0.90)
- POP: Online Structural Pruning Enables Efficient Inference of Large Foundation Models (NeurIPS 2026 · similarity 0.89)
- Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA (ICLR 2025 · similarity 0.88)
- Probe Pruning: Accelerating LLMs through Dynamic Pruning via Model-Probing (ICLR 2025 · similarity 0.88)
- Beyond Autoregression: Fast LLMs via Self-Distillation Through Time (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
