Research map / Large language models / Language, safety and interpretability
LLM pruning and scaling data
Prune LLMs after training and study data effects on scaling and collapse.
285 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2018–2026) · an approach in Language, safety and interpretability, keyword cluster “prompt · uncertainty · shot”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 2 of 5 tracked benchmarks.
77 papers in 2023–24 and 192 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 11.0% to 10.4%.
Open questions the data points to
- MMLU (accuracy): best went from 52.4 (up to 2024) to 67.75 in 2025 — From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Data (ICLR 2025 · 1 citation)
- GSM8K (accuracy): best went from 70.32 (up to 2024) to 72 in 2025 — Instruction-Following Pruning for Large Language Models (ICML 2025 · 0 citations)
What looks settled
- HellaSwag (accuracy): best up to 2024 74.3, best in 2025–26 56.22 from 3 papers, at most 0.2% better, if at all.
- PIQA (accuracy): best up to 2024 78.72, best in 2025–26 78 from 2 papers, at most 0.2% better, if at all.
- WinoGrande (accuracy): best up to 2024 69.7, best in 2025–26 67 from 2 papers, at most 0.2% better, if at all.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2018 · Breaking the Softmax Bottleneck: A High-Rank RNN Language Model (ICLR 2018 · 63 citations)
- 2019 · XLNet: Generalized Autoregressive Pretraining for Language Understanding (NeurIPS 2019 · 1,824 citations)
- 2020 · Rare Words: A Major Problem for Contextualized Embeddings and How to Fix it by Attentive Mimicking (AAAI 2020 · 85 citations)
- 2021 · COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining (NeurIPS 2021 · 128 citations)
- 2022 · TAPEX: Table Pre-training via Learning a Neural SQL Executor (ICLR 2022 · 90 citations)
- 2023 · GLM-130B: An Open Bilingual Pre-trained Model (ICLR 2023 · 293 citations)
- 2024 · A Simple and Effective Pruning Approach for Large Language Models (ICLR 2024 · 53 citations)
- 2025 · Selective Forgetting: Advancing Machine Unlearning Techniques and Evaluation in Language Models (AAAI 2025 · 12 citations)
- 2026 · Scaling Laws Revisited: Modeling the Role of Data Quality in Language Model Pretraining (ICLR 2026 · 1 citation)
Earliest papers here
- Breaking the Softmax Bottleneck: A High-Rank RNN Language Model (ICLR 2018 · 63 citations)
- XLNet: Generalized Autoregressive Pretraining for Language Understanding (NeurIPS 2019 · 1,824 citations)
- Unified Language Model Pre-training for Natural Language Understanding and Generation (NeurIPS 2019 · 941 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
MMLU · accuracy (higher is better)
7 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 52.4 | Data Engineering for Scaling Language Models to 128K Context (ICML 2024 · 4 citations) |
| 2025 | 67.75 | From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Data (ICLR 2025 · 1 citation) |
GSM8K · accuracy (higher is better)
6 papers, 2023–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 43 | Large Language Models are Human-Level Prompt Engineers (ICLR 2023 · 289 citations) |
| 2024 | 70.32 | RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models (NeurIPS 2024 · 16 citations) |
| 2025 | 72 | Instruction-Following Pruning for Large Language Models (ICML 2025 · 0 citations) |
HellaSwag · accuracy (higher is better)
5 papers, 2024–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 74.3 | PB-LLM: Partially Binarized Large Language Models (ICLR 2024 · 7 citations) |
PIQA · accuracy (higher is better)
4 papers, 2024–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 78.72 | Compressing Large Language Models by Joint Sparsification and Quantization (ICML 2024) |
WinoGrande · accuracy (higher is better)
4 papers, 2024–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 69.7 | PB-LLM: Partially Binarized Large Language Models (ICLR 2024 · 7 citations) |
What has been tried
178 papers here report 1,960 measured changes (a component added, removed or swapped, compared with a baseline): helped 66%, no clear effect 9%, hurt 19%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Compare weights per-output rather than layer-wise | WikiText validation, unstructured 50% · perplexity | 7.95 -> 7.26 | Helped | A Simple and Effective Pruning Approach for Large Language Models (ICLR 2024 · 53 citations) |
| Add support images as a demonstration example | PASCAL-5i 1-shot · mIoU | +17.3% | Helped | LLaFS: When Large Language Models Meet Few-Shot Segmentation (CVPR 2024 · 50 citations) |
| Scale model size upward | larger datasets · adjusted compression rate (%) | Helped | Language Modeling Is Compression (ICLR 2024 · 28 citations) | |
| Use positive and negative prompts instead of positive prompts only | 11-dataset one-shot classification (Avg) · accuracy | 60.1 to 61.1; 1.0% | Helped | Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · 22 citations) |
| add the sample-sample contrastive loss | In-distribution tasks (Avg) · accuracy (%) | 56.52 -> 58.54 | Helped | RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models (NeurIPS 2024 · 16 citations) |
| Initialize mask learning with a Magnitude prior instead of no prior | Wikitext · PPL | 9.12 -> 6.77 | Helped | MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models (NeurIPS 2024 · 14 citations) |
| Use offline annotated sensitive spans instead of online-selected spans | Avg. 8 classification datasets · accuracy | 41.0 -> 41.5 | Helped | Selective Forgetting: Advancing Machine Unlearning Techniques and Evaluation in Language Models (AAAI 2025 · 12 citations) |
| Add a benchmark vocabulary-size fixed effect | WikiText-103, WikiText-2, Penn Treebank · effective compute doubling time | 8.4 months -> 8.0 months | No clear effect | Algorithmic progress in language models (NeurIPS 2024 · 12 citations) |
| Replace vanilla FFN with SwiGLU | Zero-shot common-sense reasoning · average accuracy | 42.6 -> 43.9 | Helped | MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases (ICML 2024 · 11 citations) |
| Vary alpha and beta; alpha=0.1, beta=0.1 gives the lowest reported perplexity | PTB, 70% sparsity · perplexity | 18.04 -> 17.82 | Mixed | SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · 11 citations) |
| Use LLM-paraphrased queries with semantic equivariance distillation | Charades-STA SS · R@0.5 | 39.98 -> 49.09 | Helped | Omnipotent Distillation with LLMs for Weakly-Supervised Natural Language Video Localization: When Divergence Meets Consistency (AAAI 2024 · 10 citations) |
| Vary subtraction weight from λ = 1.0 to λ = 2.0 | TruthfulQA · mc1 | 35.0 -> 36.0 | Helped | Separate the Wheat from the Chaff: Model Deficiency Unlearning via Parameter-Efficient Module Operation (AAAI 2024 · 10 citations) |
What reviewers will expect
Datasets and baselines named in the 121 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- C4 · 16 of 121 papers
- MMLU · 14 of 121 papers
- GSM8K · 11 of 121 papers
- HellaSwag · 10 of 121 papers
- PIQA · 8 of 121 papers
- The Pile · 8 of 121 papers
- WinoGrande · 7 of 121 papers
- FineWeb-Edu · 7 of 121 papers
Baselines
- DSIR · 7 of 121 papers
- NPO · 7 of 121 papers
- LLM-Pruner · 5 of 121 papers
- Wanda · 5 of 121 papers
- SparseGPT · 5 of 121 papers
- DPO · 5 of 121 papers
- RMU · 5 of 121 papers
- FLAP · 4 of 121 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 269 papers; from OpenAlex and the papers' affiliations).
- Google · 17 papers
- Carnegie Mellon University · 16 papers
- Meta · 14 papers
- Tsinghua University · 11 papers
- Microsoft · 11 papers
- Stanford University · 10 papers
- Hong Kong University of Science and Technology · 9 papers
- Peking University · 8 papers
- University of Texas at Austin · 8 papers
- Google DeepMind · 7 papers
Use it
Download BibTeX of 36 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- LLM unlearning and benchmarks (298 papers)
- Controlled text generation (220 papers)
- Faithfulness of LLM explanations (169 papers)
- Data and evaluation methodology (136 papers)
- In-context learning analysis (95 papers)
- Parallel and early-exit decoding (100 papers)
Elsewhere in the topic
- Uncertainty quantification for LLMs (136 papers)
- Pruning and circuits in LLMs (135 papers)
- Statistical frameworks for LLM evaluation (124 papers)
- Safety alignment of LLMs (123 papers)
- Linguistic encoding in LLM representations (123 papers)
- Reasoning mechanisms and failure modes (114 papers)
- Speech and audio language models (111 papers)
- Sign and cross-lingual translation (96 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).