Research map / Large language models / Language, safety and interpretability
Text generation and evaluation metrics
Study embeddings, metrics, and language-model-based evaluation for natural language generation.
79 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Language, safety and interpretability, keyword cluster “natural language · speech · generation”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
21 papers in 2023–24 and 17 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 3.0% to 0.9%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Argument Mining from Speech: Detecting Claims in Political Debates (AAAI 2016 · 71 citations)
- 2017 · Data Noising as Smoothing in Neural Network Language Models (ICLR 2017 · 213 citations)
- 2018 · Learning to Predict Readability Using Eye-Movement Data From Natives and Learners (AAAI 2018 · 26 citations)
- 2019 · code2seq: Generating Sequences from Structured Representations of Code (ICLR 2019)
- 2020 · MaskGEC: Improving Neural Grammatical Error Correction via Dynamic Masking (AAAI 2020 · 64 citations)
- 2021 · KLUE: Korean Language Understanding Evaluation (NeurIPS 2021 · 76 citations)
- 2022 · Synchromesh: Reliable Code Generation from Pre-trained Language Models (ICLR 2022 · 62 citations)
- 2023 · Multi-lingual Evaluation of Code Generation Models (ICLR 2023 · 26 citations)
- 2024 · Chinese Spelling Correction as Rephrasing Language Model (AAAI 2024 · 22 citations)
- 2025 · LLM-RG4: Flexible and Factual Radiology Report Generation Across Diverse Input Contexts (AAAI 2025 · 13 citations)
- 2026 · ViDia2Std: A Parallel Corpus and Methods for Low-Resource Vietnamese Dialect-to-Standard Translation (AAAI 2026 · 2 citations)
Earliest papers here
- Argument Mining from Speech: Detecting Claims in Political Debates (AAAI 2016 · 71 citations)
- A Morphology-Aware Network for Morphological Disambiguation (AAAI 2016 · 20 citations)
- Data Noising as Smoothing in Neural Network Language Models (ICLR 2017 · 213 citations)
What has been tried
62 papers here report 636 measured changes (a component added, removed or swapped, compared with a baseline): helped 66%, no clear effect 8%, hurt 22%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Mask only non-error characters rather than any characters for auxiliary MLM | ECSpell average · F1 | 85.8 -> 88.1 | Helped | Chinese Spelling Correction as Rephrasing Language Model (AAAI 2024 · 22 citations) |
| Add TLW | MIMIC-RG4, stage 2 average · F1-14 | 0.551 -> 0.577 | Helped | LLM-RG4: Flexible and Factual Radiology Report Generation Across Diverse Input Contexts (AAAI 2025 · 13 citations) |
| Generate target structures with the LLM rather than use sampled human-annotated structures | ACE05, k=10, N=10 · Argument classification F1 | 41.77 -> 45.73 | Helped | STAR: Boosting Low-Resource Information Extraction by Structure-to-Text Data Generation with Large Language Models (AAAI 2024 · 12 citations) |
| Use LoRA instead of Prefix Tuning | 13 languages · ParaScore | 77.45±8.5 -> 90.67±2.5 | Helped | LAMPAT: Low-Rank Adaption for Multilingual Paraphrasing Using Adversarial Training (AAAI 2024 · 8 citations) |
| add two related words to each label name | FEW-NERD · Loose Micro-F1 | 85.2 -> 85.6 | Helped | Labels Need Prompts Too: Mask Matching for Natural Language Understanding Tasks (AAAI 2024 · 4 citations) |
| Add swap augmentation | Vietnamese-Bahnar · BLEU | 49.20 -> 52.94 | Helped | Revitalizing Bahnaric Language through Neural Machine Translation: Challenges, Strategies, and Promising Outcomes (AAAI 2024 · 1 citation) |
| Increase the annotator-consistency threshold k from 0 to 0.45 | Toxicity Ratings (TR) and VOICED, political-group offense modeling · F1-score | Helped | ARTICLE: Annotator Reliability Through In-Context Learning (AAAI 2025 · 0 citations) | |
| add semantic invariance loss | ASRU 2019 Dev · MER | 10.91 -> 10.55 | Helped | Code-switching Mediated Sentence-level Semantic Learning (AAAI 2025 · 0 citations) |
| Use joint multi-task training instead of single-task training | TiALD test, comment only · TiALD Score | 64.57 -> 64.81 | Helped | A Multi-Task Benchmark for Abusive Language Detection in Low-Resource Settings (NeurIPS 2025 · 0 citations) |
| replace simple pooling with average pooling | 1B, 60B tokens; Hellaswag · accuracy | 62.9 ± 0.9 -> 62.5 ± 1.0 | No clear effect | From Bytes to Ideas: Language Modeling with Autoregressive U-Nets (NeurIPS 2025 · 0 citations) |
| Replace the LSTM-predicted bias with previous-year word frequency | ACL abstracts 2021 test · PPL | 20.20 -> 21.23 | Hurt | Future Language Modeling from Temporal Document History (ICLR 2024 · 0 citations) |
| Use Cosine rather than Euclidean distance for implicitness scoring | ImpScore test set · Implicitness Accuracy | Helped | ImpScore: A Learnable Metric For Quantifying The Implicitness Level of Sentences (ICLR 2025) |
What reviewers will expect
Datasets and baselines named in the 10 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Baselines
- GPT-4o · 2 of 10 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 38 papers; from OpenAlex and the papers' affiliations).
- Microsoft · 2 papers
- Amazon · 2 papers
- University of Illinois, Urbana Champaign · 2 papers
- Shanghai Jiao Tong University · 2 papers
- Ho Chi Minh City University of Technology · 2 papers
- Tsinghua University · 2 papers
- Korea Advanced Institute of Science and Technology · 2 papers
- ETHZ - ETH Zurich · 2 papers
- Swiss Federal Institute of Technology · 2 papers
Use it
Download BibTeX of 39 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Speech and audio language models (111 papers)
- Multilingual LLM data and culture (69 papers)
- Sign language datasets and translation (23 papers)
- Sign and cross-lingual translation (96 papers)
Elsewhere in the topic
- LLM unlearning and benchmarks (298 papers)
- LLM pruning and scaling data (285 papers)
- Controlled text generation (220 papers)
- Faithfulness of LLM explanations (169 papers)
- Data and evaluation methodology (136 papers)
- Uncertainty quantification for LLMs (136 papers)
- Pruning and circuits in LLMs (135 papers)
- Statistical frameworks for LLM evaluation (124 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).