Research map / Large language models / Language, safety and interpretability
Sign and cross-lingual translation
Translates sign and spoken languages using large language models and multilingual pretraining.
96 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2018–2026) · an approach in Language, safety and interpretability, keyword cluster “natural language · speech · generation”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
25 papers in 2023–24 and 30 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 3.6% to 1.6%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2018 · Deep contextualized word representations (ICLR 2018 · 12,263 citations)
- 2019 · Unsupervised Multi-Modal Neural Machine Translation (CVPR 2019 · 72 citations)
- 2020 · ERNIE 2.0: A Continual Pre-Training Framework for Language Understanding (AAAI 2020 · 745 citations)
- 2021 · How2Sign: A Large-Scale Multimodal Dataset for Continuous American Sign Language (CVPR 2021 · 256 citations)
- 2022 · A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation (CVPR 2022 · 158 citations)
- 2023 · Natural Language-Assisted Sign Language Recognition (CVPR 2023 · 106 citations)
- 2024 · SALMONN: Towards Generic Hearing Abilities for Large Language Models (ICLR 2024 · 20 citations)
- 2025 · Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues (CVPR 2025 · 13 citations)
- 2026 · SpeakerLM: End-to-End Versatile Speaker Diarization and Recognition with Multimodal Large Language Models (AAAI 2026 · 2 citations)
Earliest papers here
- Deep contextualized word representations (ICLR 2018 · 12,263 citations)
- Learning Better Name Translation for Cross-Lingual Wikification (AAAI 2018 · 13 citations)
- Cross-Lingual Entity Linking for Web Tables (AAAI 2018 · 12 citations)
What has been tried
83 papers here report 977 measured changes (a component added, removed or swapped, compared with a baseline): helped 74%, no clear effect 6%, hurt 18%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add activation tuning using stories | WikiQA · Accuracy (FR) | 0.19 (0.29) -> 0.41 (0.98) | Helped | SALMONN: Towards Generic Hearing Abilities for Large Language Models (ICLR 2024 · 20 citations) |
| Use vector-quantized neural spectrum prediction rather than directly predicting tokenizer embeddings | TUEV · Weighted F1 | 0.8170±0.0058 -> 0.8312±0.0052 | Helped | Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI (ICLR 2024 · 19 citations) |
| Add pseudo-glosses to visual features | BOBSL SENT-VAL · BLEURT | 41.0 -> 41.8 | Helped | Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues (CVPR 2025 · 13 citations) |
| use HuBERT L9 rather than HuBERT avg as the semantic teacher | SLMTokBench · MI | 30.9 -> 32.9 | Helped | SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models (ICLR 2024 · 12 citations) |
| add utterance intent-slot interaction | MixATIS · Overall accuracy | decrease of 1.4% | Helped | Uni-MIS: United Multiple Intent Spoken Language Understanding via Multi-View Intent-Slot Interaction (AAAI 2024 · 11 citations) |
| Replace VAC with the cleaned baseline by removing its alignment constraint and last temporal pooling layer | PHOENIX14 · WER | 21.2 -> 20.7; 22.3 -> 21.1 | Helped | Cross-Sentence Gloss Consistency for Continuous Sign Language Recognition (AAAI 2024 · 8 citations) |
| Increase SLT pretraining data from YT-ASL to YT-Full | How2Sign and FLEURS-ASL#0 (En) · BLEURT | gain reaches ~7 BLEURT on How2Sign and ~11 BLEURT on FLEURS-ASL#0 (En) for Base and XL, respectively | Helped | Scaling Sign Language Translation (NeurIPS 2024 · 8 citations) |
| Apply LoRA to convolution, query, key, and value weights rather than query, key, and value only | VoxLRS-SA, S1 · WER (%) | 38.5 -> 38.6 | Hurt | Personalized Lip Reading: Adapting to Your Unique Lip Movements with Vision and Language (AAAI 2025 · 6 citations) |
| Apply SignCL during pretraining only | PHOENIX-2014T test · B@4 | 21.25 -> 22.69 | Helped | Improving Gloss-free Sign Language Translation by Reducing Representation Density (NeurIPS 2024 · 6 citations) |
| Add spatial adapters | Phoenix14T dev · BLEU4 | 16.38 -> 19.55 | Helped | Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation (ICLR 2024 · 5 citations) |
| Add text correction to KD-MSLR | PHOENIX14 Dev WER · WER | 28.7 -> 17.4 | Helped | KD-MSLRT: Lightweight Sign Language Recognition Model Based on Mediapipe and 3D to 1D Knowledge Distillation (AAAI 2025 · 4 citations) |
| Joint omni-modal alignment (image-text plus speech-unit-text trained together) instead of single bi-modal alignment | vision-language and LibriSpeech ASR · vision-language benchmarks and 1-WER | Helped | EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions (CVPR 2025 · 4 citations) |
What reviewers will expect
Datasets and baselines named in the 22 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- LibriSpeech · 4 of 22 papers
- PHOENIX14T · 2 of 22 papers
- COCO · 2 of 22 papers
- GRID · 2 of 22 papers
- LibriTTS · 2 of 22 papers
- AISHELL-1 · 2 of 22 papers
- BOBSL · 2 of 22 papers
- How2Sign · 2 of 22 papers
Baselines
- BERT · 3 of 22 papers
- RoBERTa · 2 of 22 papers
- GPT-4o · 2 of 22 papers
- Sincan et al. · 2 of 22 papers
- GFSLT · 2 of 22 papers
- Sign2GPT · 2 of 22 papers
- SSLT · 2 of 22 papers
- SSVP-SLT · 2 of 22 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 55 papers; from OpenAlex and the papers' affiliations).
- Microsoft Research Asia · 4 papers
- Tsinghua University · 4 papers
- Shanghai Jiao Tong University · 3 papers
- Tencent · 3 papers
- Alibaba Group · 3 papers
- University of Technology Sydney · 3 papers
- Hong Kong University of Science and Technology · 3 papers
- Microsoft · 3 papers
- Meta · 3 papers
- The Chinese University of Hong Kong · 3 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Text generation and evaluation metrics (79 papers)
- Speech and audio language models (111 papers)
- Multilingual LLM data and culture (69 papers)
- Sign language datasets and translation (23 papers)
Elsewhere in the topic
- LLM unlearning and benchmarks (298 papers)
- LLM pruning and scaling data (285 papers)
- Controlled text generation (220 papers)
- Faithfulness of LLM explanations (169 papers)
- Data and evaluation methodology (136 papers)
- Uncertainty quantification for LLMs (136 papers)
- Pruning and circuits in LLMs (135 papers)
- Statistical frameworks for LLM evaluation (124 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).