Research map / Label-efficient and robust learning / Self-supervised and contrastive
Self-supervised speech and brain decoding
Uses wav2vec-style speech models to model and decode brain activity.
56 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Self-supervised and contrastive, keyword cluster “depth · self supervision · pre”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
12 papers in 2023–24 and 17 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 3.7% to 5.3%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Look Into Person: Self-Supervised Structure-Sensitive Learning and a New Benchmark for Human Parsing (CVPR 2017 · 532 citations)
- 2020 · PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization (ICML 2020 · 975 citations)
- 2021 · Learning an Effective Context-Response Matching Model with Self-Supervised Tasks for Retrieval-based Dialogues (AAAI 2021 · 49 citations)
- 2022 · SSAST: Self-Supervised Audio Spectrogram Transformer (AAAI 2022 · 281 citations)
- 2023 · Self-Supervised Implicit Glyph Attention for Text Recognition (CVPR 2023 · 28 citations)
- 2024 · Self-Supervised Disentangled Representation Learning for Robust Target Speech Extraction (AAAI 2024 · 5 citations)
- 2025 · Distilling Structural Representations into Protein Sequence Models (ICLR 2025 · 11 citations)
- 2026 · mRNABench: A curated benchmark for mature mRNA property and function prediction (NeurIPS 2026 · 3 citations)
Earliest papers here
- Look Into Person: Self-Supervised Structure-Sensitive Learning and a New Benchmark for Human Parsing (CVPR 2017 · 532 citations)
- PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization (ICML 2020 · 975 citations)
- wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (NeurIPS 2020 · 400 citations)
What has been tried
45 papers here report 523 measured changes (a component added, removed or swapped, compared with a baseline): helped 69%, no clear effect 5%, hurt 22%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add MFE and SS auxiliary losses | Muscle-TE · Spearman rank | 0.550 -> 0.573 | Helped | mRNA2vec: mRNA Embedding with Language Model in the 5'UTR-CDS for mRNA Design (AAAI 2025 · 6 citations) |
| add instance normalization in E_c | WSJ0-2mix-extr · SI-SNRi | 18.2 -> 19.9 | Helped | Self-Supervised Disentangled Representation Learning for Robust Target Speech Extraction (AAAI 2024 · 5 citations) |
| Share model parameters across modalities rather than train separate models | LRS3 full-data, VSR and AVSR · WER (%) | VSR: 85.5 -> 36.4; AVSR: 63.4 -> 2.1 | Helped | Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · 4 citations) |
| Use average-token semantics rather than CLS-token semantics | VOCASET · MVE | Mixed | Wav2Sem: Plug-and-Play Audio Semantic Decoupling for 3D Speech-Driven Facial Animation (CVPR 2025 · 2 citations) | |
| Add masked gene prediction loss | Novel cell type classification Dct1 · Accuracy | 0.287 -> 0.335 | Helped | Cell ontology guided transcriptome foundation model (NeurIPS 2024 · 2 citations) |
| add self-supervised pretext-task pre-training | Heard speech MEG · ROC AUC | .519 (± .002) -> .630 (± .000) | Helped | The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning (ICML 2025 · 1 citation) |
| Add BSQ rather than use FSQ | LibriSpeech test-clean · dWER | 2.61 -> 2.54 | Helped | FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks (NeurIPS 2025 · 1 citation) |
| Remove commitment loss | Amazon-Sports · perplexity of previously learned IDs | 305.39 -> 147.10 | Hurt | Language Models as Semantic Indexers (ICML 2024 · 1 citation) |
| replace 5% of MSA tokens with random tokens | CASP14 · LDDT | 53.2 -> 50.6 | Hurt | MSA Generation with Seqs2Seqs Pretraining: Advancing Protein Structure Predictions (NeurIPS 2024 · 1 citation) |
| Add the common representation | Cambridge · accuracy | 94.44 -> 95.56 | Helped | Self-Supervised Collaborative Information Bottleneck for Text Readability Assessment (AAAI 2025 · 0 citations) |
| Use cfg 2.0 instead of cfg 0.0 | VCTK · SIM | 0.50 -> 0.55 | Helped | Metis: A Foundation Speech Generation Model with Masked Generative Pre-training (NeurIPS 2025 · 0 citations) |
| calculate the VQScore with cosine similarity in code space instead of signal space | VoiceBank-DEMAND noisy test · correlation with PESQ | 0.6514 -> 0.7941 | Helped | Self-Supervised Speech Quality Estimation and Enhancement Using Only Clean Speech (ICLR 2024 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 11 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- LibriTTS · 2 of 11 papers
- LibriMix · 2 of 11 papers
- WHAM! · 2 of 11 papers
- VCTK · 2 of 11 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 29 papers; from OpenAlex and the papers' affiliations).
- Mila · 3 papers
- The Chinese University of Hong Kong · 2 papers
- University of Montreal · 2 papers
- Cornell University · 2 papers
Use it
Download BibTeX of 39 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Pretext-task visual pre-training (136 papers)
- Self-supervision in supervised pipelines (121 papers)
- Self-supervised neuroimaging representations (98 papers)
Elsewhere in the topic
- Invariance-based self-supervised learning (168 papers)
- Data-efficient and robust self-supervision (122 papers)
- Contrastive versus non-contrastive theory (74 papers)
- Inductive bias and minimal representations (66 papers)
- Masked autoencoder pretraining (48 papers)
- Empirical contrastive visual learning (38 papers)
- Representation evaluation and trade-offs (37 papers)
- Self-supervised speech representations (24 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).