Research map / Label-efficient and robust learning / Recognition and long-tail
Multilingual language representation pretraining
Pretrains cross-lingual language models and augmentation for transferable text representations.
507 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Recognition and long-tail, keyword cluster “representations · transfer · unsupervised”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Leveling off Several papers from 2025–26 still report on 1 tracked benchmark, but the best result has barely moved since 2024.
110 papers in 2023–24 and 65 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 8.8% to 5.0%.
What looks settled
- MR (accuracy): best up to 2024 82.5, best in 2025–26 69.87 from 2 papers, at most 0.2% better, if at all.
No longer reported by papers here: WMT14 En-De (BLEU, last in 2023); GENIA (F1, last in 2023); MSRA (F1, last in 2024); RTE (accuracy, last in 2024); Resume (F1, last in 2024).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings (NeurIPS 2016 · 1,340 citations)
- 2017 · A Simple but Tough-to-Beat Baseline for Sentence Embeddings (ICLR 2017 · 1,038 citations)
- 2018 · Word translation without parallel data (ICLR 2018 · 635 citations)
- 2019 · Cross-lingual Language Model Pretraining (NeurIPS 2019 · 1,596 citations)
- 2020 · PHASEN: A Phase-and-Harmonics-Aware Speech Enhancement Network (AAAI 2020 · 327 citations)
- 2021 · A Joint Training Dual-MRC Framework for Aspect Based Sentiment Analysis (AAAI 2021 · 215 citations)
- 2022 · Unified Named Entity Recognition as Word-Word Relation Classification (AAAI 2022 · 392 citations)
- 2023 · Robust Speech Recognition via Large-Scale Weak Supervision (ICML 2023 · 1,168 citations)
- 2024 · DePT: Decoupled Prompt Tuning (CVPR 2024 · 50 citations)
- 2025 · Phoneme-Level Feature Discrepancies: A Key to Detecting Sophisticated Speech Deepfakes (AAAI 2025 · 12 citations)
- 2026 · BLOCK-EM: Preventing Emergent Misalignment via Latent Blocking (ICML 2026 · 0 citations)
Earliest papers here
- Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings (NeurIPS 2016 · 1,340 citations)
- Dual Learning for Machine Translation (NeurIPS 2016 · 720 citations)
- Gated Neural Networks for Targeted Sentiment Analysis (AAAI 2016 · 311 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
WMT14 En-De · BLEU (higher is better)
7 papers, 2019–2023 · not reported since 2023
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 27.73 | Addressing the Under-Translation Problem from the Entropy Perspective (AAAI 2019 · 13 citations) |
| 2020 | 30.75 | Incorporating BERT into Neural Machine Translation (ICLR 2020 · 170 citations) |
| 2021 | 31.74 | On Learning Universal Representations Across Languages (ICLR 2021 · 65 citations) |
GENIA · F1 (higher is better)
5 papers, 2020–2023 · not reported since 2023
| Year | Best so far | Set by |
|---|---|---|
| 2020 | 79.8 | Recursively Binary Modification Model for Nested Named Entity Recognition (AAAI 2020 · 5 citations) |
| 2022 | 81.39 | Unified Named Entity Recognition as Word-Word Relation Classification (AAAI 2022 · 392 citations) |
| 2023 | 82.09 | Nested Named Entity Recognition as Building Local Hypergraphs (AAAI 2023 · 20 citations) |
MR · accuracy (higher is better)
5 papers, 2017–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2017 | 81.06 | Neural Bag-of-Ngrams (AAAI 2017 · 9 citations) |
| 2018 | 82.5 | Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (ICLR 2018 · 99 citations) |
MSRA · F1 (higher is better)
5 papers, 2021–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 96.41 | Dynamic Modeling Cross- and Self-Lattice Attention Network for Chinese NER (AAAI 2021 · 28 citations) |
RTE · accuracy (higher is better)
5 papers, 2021–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 72.6 | Revisiting Few-sample BERT Fine-tuning (ICLR 2021 · 210 citations) |
| 2024 | 85.9 | Sparse is Enough in Fine-tuning Pre-trained Large Language Models (ICML 2024 · 1 citation) |
Resume · F1 (higher is better)
5 papers, 2021–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 96.67 | Dynamic Modeling Cross- and Self-Lattice Attention Network for Chinese NER (AAAI 2021 · 28 citations) |
| 2023 | 97.33 | SSMI: Semantic Similarity and Mutual Information Maximization Based Enhancement for Chinese NER (AAAI 2023 · 10 citations) |
What has been tried
453 papers here report 4,928 measured changes (a component added, removed or swapped, compared with a baseline): helped 72%, no clear effect 6%, hurt 19%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add CAT to the ITM head, without test-time fusion | Base-to-new, 11-dataset average · H | 75.18 -> 76.63 | Helped | DePT: Decoupled Prompt Tuning (CVPR 2024 · 50 citations) |
| add semantic contrastive learning | XTREME-Avg · F1 | Helped | MCL-NER: Cross-Lingual Named Entity Recognition via Multi-View Contrastive Learning (AAAI 2024 · 33 citations) | |
| Use heterogeneous emotion, intensity, speaker, and audio nodes instead of omitting emotion nodes | DailyTalk test · E-DMOS | 3.496 -> 3.619 | Helped | Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context Modeling (AAAI 2024 · 24 citations) |
| Add Syntax Distance Weighting | Rest.14 · Implicit accuracy | 2.62↓ | Helped | Aspect-Based Sentiment Analysis with Explicit Sentiment Augmentations (AAAI 2024 · 23 citations) |
| Use adaptive rather than fixed margin | NISQA TEST FOR · PC | 0.96 -> 0.97 | No clear effect | SCOREQ: Speech Quality Assessment with Contrastive Regression (NeurIPS 2024 · 15 citations) |
| add GAT to model the temporal dependencies of phoneme-level features | InTheWild · EER (%) | 28.66 -> 16.07 | Helped | Phoneme-Level Feature Discrepancies: A Key to Detecting Sophisticated Speech Deepfakes (AAAI 2025 · 12 citations) |
| Add the information bottleneck module | REST14 · accuracy | 85.54 -> 87.77 | Helped | Counterfactual-Enhanced Information Bottleneck for Aspect-Based Sentiment Analysis (AAAI 2024 · 11 citations) |
| add quantum projection to ACS | SemEval-2016 average (FR, SP, DU, RU) · F1-score | 53.23 -> 54.58 | Helped | QPEN: Quantum Projection and Quantum Entanglement Enhanced Network for Cross-Lingual Aspect-Based Sentiment Analysis (AAAI 2024 · 10 citations) |
| Add soft prompt | CoNLL03 shot-20 · F1 | 80.5 -> 81.9 | Helped | RoPDA: Robust Prompt-Based Data Augmentation for Low-Resource Named Entity Recognition (AAAI 2024 · 8 citations) |
| Add span enhancement with convolution-based n-gram features | Resume · F1 | 96.26 -> 96.86 | Helped | SENCR: A Span Enhanced Two-Stage Network with Counterfactual Rethinking for Chinese NER (AAAI 2024 · 8 citations) |
| Add multi-view word-sense representations | SE2 · F1-score | 79.4 -> 80.8 | Helped | Quantum Interference Model for Semantic Biases of Glosses in Word Sense Disambiguation (AAAI 2024 · 7 citations) |
| Add Ext-Doc objective | CNNDM · R1 | 40.7 -> 40.9 | Helped | Unsupervised Extractive Summarization with Learnable Length Control Strategies (AAAI 2024 · 7 citations) |
What reviewers will expect
Datasets and baselines named in the 42 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- LibriSpeech · 3 of 42 papers
- Twitter · 2 of 42 papers
- MR · 2 of 42 papers
- Snippets · 2 of 42 papers
- VCTK · 2 of 42 papers
Baselines
- BERT · 3 of 42 papers
- BM25 · 3 of 42 papers
- RoBERTa · 3 of 42 papers
- GPT-3.5 · 2 of 42 papers
- TF-IDF+SVM · 2 of 42 papers
- PTE · 2 of 42 papers
- CNN · 2 of 42 papers
- LSTM · 2 of 42 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 175 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 8 papers
- Zhejiang University · 7 papers
- Microsoft · 7 papers
- Chinese Academy of Sciences · 7 papers
- Google · 7 papers
- Carnegie Mellon University · 6 papers
- National University of Singapore · 5 papers
- University of Chinese Academy of Sciences · 5 papers
- Harbin Institute of Technology · 5 papers
- Meta · 5 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Contrastive representation theory (495 papers)
- Correspondence and causal representation learning (447 papers)
- Dense features and foundation priors (482 papers)
- Data augmentation and generalization bias (409 papers)
- Recommendation representation learning (207 papers)
Elsewhere in the topic
- Partial and multi-label learning (345 papers)
- Loss design and classifier theory (273 papers)
- Group-robust classification (251 papers)
- Learning from unlabeled and positive-unlabeled data (240 papers)
- Confidence calibration objectives (129 papers)
- Margin-based face recognition losses (124 papers)
- Supervised person re-identification (108 papers)
- Self-explaining and visual explanation networks (108 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).