Research map / Label-efficient and robust learning / Recognition and long-tail
Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add supervised contrastive learning to cross-entropy fine-tuning | RoBERTa-Large · GLUE few-shot (20 examples; SST-2, QNLI, MNLI) | accuracy: 10.7 points improvement on QNLI, 3.4 points improvement on MNLI, and 2.2 points improvement on SST-2 | Helped |
| Add supervised contrastive learning to cross-entropy fine-tuning | RoBERTa-Large · GLUE few-shot (100 examples; MNLI) | accuracy: 1.9 points improvement | Helped |
| Add supervised contrastive learning to cross-entropy fine-tuning | RoBERTa-Large · GLUE few-shot (1000 examples; QNLI) | accuracy: 0.6 points improvement | Helped |
| Add supervised contrastive learning to cross-entropy fine-tuning | RoBERTa-Large · GLUE few-shot (1000 examples; SST-2 and MNLI) | accuracy | No clear effect |
| Add supervised contrastive learning to cross-entropy fine-tuning | RoBERTa-Large · Augmented noisy GLUE training sets (100 examples; SST-2, QNLI, MNLI) | accuracy: 0.4 points improvement on SST-2, 2.5 points improvement on QNLI, and 7 points improvement on MNLI | Helped |
| Add supervised contrastive learning to cross-entropy fine-tuning | RoBERTa-Large · Noisy MNLI (T=0.7) | accuracy: 7.7 points improvement | Helped |
| Add supervised contrastive learning to cross-entropy fine-tuning | RoBERTa-Large · Noisy QNLI (T=0.9) | accuracy: 4.2 points improvement | Helped |
| Add supervised contrastive learning to cross-entropy fine-tuning | RoBERTa-Large · GLUE full dataset (6 datasets) | accuracy: 3.1 points improvement on MRPC, 3.5 points improvement on QNLI, and an average improvement of 1.2 points across all 6 da | Helped |
| Add supervised contrastive learning to cross-entropy fine-tuning | RoBERTa-Large · GLUE full dataset (other datasets besides MRPC and QNLI) | accuracy | No clear effect |
| Use CE+SCL for source-task and target-task fine-tuning | RoBERTa-Large transferred from SST-2 · Amazon-2 (20 labeled examples per class) | accuracy: 2.9 point improvement | Helped |
| Use CE+SCL for source-task and target-task fine-tuning | RoBERTa-Large transferred from SST-2 · Yelp-2 (20 labeled examples per class) | accuracy: 0.4 points improvement | Helped |
| Increase batch size from 16 to 256 (ablation) | RoBERTa-Base CE+SCL · Full-dataset SST-2, CoLA, QNLI, and MNLI | accuracy: 0.3 points improvement on SST-2, 0.8 points improvement on CoLA, 0.4 points improvement on QNLI, and 1.3 points improvem | Helped |
Closest papers by meaning
- Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data (ICML 2025 · similarity 0.88)
- Understanding Linear Probing then Fine-tuning Language Models from NTK Perspective (NeurIPS 2024 · similarity 0.87)
- FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain (ICML 2025 · similarity 0.87)
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.87)
- Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning (NeurIPS 2025 · similarity 0.87)
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.