Research map / Label-efficient and robust learning / Self-supervised and contrastive
A Mutual Information Maximization Perspective of Language Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the DIM sentence-to-n-gram contrastive objective (ablation) | BERT-NCE with BERT Base · GLUE | average: 78.2 -> 78.9 | Helped |
| Add the DIM sentence-to-n-gram contrastive objective (ablation) | BERT-NCE with BERT Large · GLUE | average: 80.6 -> 81.1 | Helped |
| Add the DIM sentence-to-n-gram contrastive objective (ablation) | BERT-NCE with BERT Base · SQuAD 1.1 Dev | F1: 90.2 -> 90.7 | Helped |
| Add the DIM sentence-to-n-gram contrastive objective (ablation) | BERT-NCE with BERT Base · SQuAD 1.1 Dev | EM: 83.3 -> 84.0 | Helped |
| Add the DIM sentence-to-n-gram contrastive objective (ablation) | BERT-NCE with BERT Large · SQuAD 1.1 Dev | F1: 92.0 -> 92.6 | Helped |
| Add the DIM sentence-to-n-gram contrastive objective (ablation) | BERT-NCE with BERT Large · SQuAD 1.1 Dev | EM: 85.9 -> 86.6 | Helped |
| Add the DIM sentence-to-n-gram contrastive objective (ablation) | BERT-NCE with BERT Base · SQuAD 1.1 Test | F1: 90.9 -> 91.4 | Helped |
| Add the DIM sentence-to-n-gram contrastive objective (ablation) | BERT-NCE with BERT Base · SQuAD 1.1 Test | EM: 84.4 -> 84.7 | Helped |
| Add the DIM sentence-to-n-gram contrastive objective (ablation) | BERT-NCE with BERT Large · SQuAD 1.1 Test | F1: 92.7 -> 93.1 | Helped |
| Add the DIM sentence-to-n-gram contrastive objective (ablation) | BERT-NCE with BERT Large · SQuAD 1.1 Test | EM: 86.6 -> 87.3 | Helped |
| Use the BERT-NCE reimplementation: MLM only, negative sampling, and one sentence per training example | BERT with BERT Base · GLUE | average: 78.8 -> 78.2 | Hurt |
| Use the BERT-NCE reimplementation: MLM only, negative sampling, and one sentence per training example | BERT with BERT Base · SQuAD 1.1 Dev | F1: 88.5 -> 90.2 | Helped |
Closest papers by meaning
- Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models (NeurIPS 2025 · similarity 0.87)
- Attention with Trained Embeddings Provably Selects Important Tokens (NeurIPS 2025 · similarity 0.87)
- Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe (NeurIPS 2024 · similarity 0.86)
- An Information-theoretic Multi-task Representation Learning Framework for Natural Language Understanding (AAAI 2025 · similarity 0.86)
- Narrowing the Gap between Supervised and Unsupervised Sentence Representation Learning with Large Language Model (AAAI 2024 · similarity 0.86)
- Layer by Layer: Uncovering Hidden Representations in Language Models (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.