AI Paper Atlas

Research map / Architectures and efficiency / Pruning and efficient tuning

Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers

Sotiris Anagnostidis, Dario Pavllo, Luca Biggio, Lorenzo Noci, Aurelien Lucchi, Thomas Hofmann

NeurIPS 2023 · Architectures and efficiency › Pruning and efficient tuning · approach: Layer-wise pruning of LLMs · 10 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add learned dynamic context pruning (~80% sparsity)GPT-2-small · Wikipedia+BookCorpus context 1000perplexity: -0.085No clear effect
add dynamic context pruning instead of local or sparse attention at same sparsityGPT-2-small · Wikipedia+BookCorpusperplexityImproved
add dynamic pruning to GPT-2-small for long context throughputGPT-2-small · long contextthroughput: +98% throughput for 0.316 perplexity lossImproved
add dynamic pruning to GPT-2-mediumGPT-2-medium · context 1000throughput: +189% throughput for 0.084 perplexity lossImproved
train only interaction parameters, freeze rest of network (ablation)dynamic context pruning · Wikipedia+BookCorpusperplexity: +9.285 perplexityHurt
tie dropping decisions across depth (ablation)dynamic context pruning · GPT-2 modelsperplexityHurt
use regular sigmoid with temperature instead of alpha-sigmoid (ablation)dynamic context pruning · GPT-2-smalltraining stabilityHurt
increase interaction dimension r (ablation)dynamic context pruning · GPT-2-smallsparsity at fixed perplexityImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy