AI Paper Atlas

Research map / Architectures and efficiency / Pruning and efficient tuning

Discovering Salient Neurons in deep NLP models

Nadir Durrani, Fahim Dalvi, Hassan Sajjad

JMLR 2023 · Architectures and efficiency › Pruning and efficient tuning · approach: Parameter-efficient transfer analysis · 16 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
keep LCA top 20% neurons instead of random 20% (ablation)BERT · POSaccuracy: 28.45 -> 90.16Improved
keep LCA top 20% neurons instead of bottom 20% (ablation)XLNet · SEMaccuracy: 25.37 -> 90.70Improved
rank neurons by LCA instead of weight magnitude (ablation)RoBERTa · POSaccuracy: 71.06 -> 91.93Improved
rank neurons by LCA instead of random selection (ablation)BERT · Chunkingaccuracy: 75.83 -> 89.01Improved
retrain classifier on top 5% neurons instead of all neuronsXLNet · POSaccuracy: 96.19 -> 96.49No clear effect
rank neurons with LCA instead of Probeless (ablation)BERT · Chunking (10% neurons)accuracy: 90.36 -> 94.99Improved
rank neurons with LCA instead of Probeless (ablation)BERT · POS (5% neurons)accuracy: 95.45 -> 95.92No clear effect
select top neurons only instead of all neuronsBERT · POSselectivity: 14.49 -> 31.84Improved
retrain on random neurons instead of top neurons (ablation)BERT · POS (5% neurons)accuracy: 95.92 -> 94.12Hurt
retrain on bottom neurons instead of top neurons after QNLI fine-tuningXLNet · POSaccuracy: 96.49 -> 10.39Hurt
fine-tune on GLUE task (MRPC) before probing bottom neuronsBERT · POSaccuracy: 93.20 -> 93.13No clear effect
use top neurons in multilingual XLM-R instead of all neuronsXLM-R · POS Englishaccuracy: 94.70 -> 94.17No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy