Research map / Large language models / Language, safety and interpretability
Narrowing the Gap between Supervised and Unsupervised Sentence Representation Learning with Large Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Explains the supervised vs. unsupervised contrastive sentence embedding gap via training-data similarity-pattern complexity (Relative Fitting Difficulty), then narrows it by using GPT-3.5 in-context learning to generate STS/NLI-style pattern data plus a Hierarchical Triplet loss.
Problem
The large STS performance gap between supervised and unsupervised contrastive sentence embedding (CSE) training, where the only difference is training data; prior work explained it only via alignment and uniformity, which describe results rather than the causal training-data factors.
Approach
Identifies the similarity pattern of training data as the key factor, defines Relative Fitting Difficulty (RFD) as the gap in alignment/uniformity between held-out training data and evaluation data, and uses LLM in-context learning to simulate STS and NLI similarity patterns (with a hierarchical intermediate sentence and a Hierarchical Triplet loss) to raise pattern complexity of unsupervised data
Key results
- With BERT-base, Wiki.STS_HT reaches 79.79 average Spearman on STS tasks and NLI.STS_HT reaches 80.02, outperforming all unsupervised baselines (best unsupervised baseline InfoCSE 78.85) but below supervised PromptBERT (81.97).
- With RoBERTa-base, NLI.STS_HT reaches 81.33 average Spearman on STS, versus 79.15 for PromptRoBERTa unsupervised and 82.95 for supervised PromptRoBERTa.
- Removing HT loss and using only contrastive loss with multiple positives gives 79.14, below the HT loss result of 79.79 on Wiki.STS_HT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| introduce LLM-generated complex similarity patterns into unsupervised training data | U-CSE · STS tasks | Spearman's correlation | Helped |
| use LLM-simulated hierarchical STS patterns with Hierarchical Triplet loss instead of ordinary unsupervised training | U-CSE · STS tasks | Spearman's correlation | Helped |
| increase the number of LLM-generated training examples (ablation) | Wiki.STS_HT with BERT-base · STS tasks (Avg.) | Spearman's correlation | Helped |
| invert HT margins to use a large m1 and small m2 (ablation) | Wiki.STS_HT with BERT-base · STS tasks (Avg.) | Spearman's correlation | Hurt |
| replace HT loss with contrastive loss using only one positive sentence (ablation) | Wiki.STS with BERT-base · STS tasks (Avg.) | Spearman's correlation: 79.79$_\mathrm{\pm 0.23}$ -> 78.72$_\mathrm{\pm 0.24}$ | Hurt |
| replace HT loss with contrastive loss using multiple positive sentences (ablation) | Wiki.STS with BERT-base · STS tasks (Avg.) | Spearman's correlation: 79.79$_\mathrm{\pm 0.23}$ -> 79.14$_\mathrm{\pm 0.27}$ | Hurt |
| add an intermediate positive sentence while retaining contrastive loss (ablation) | Wiki.STS with BERT-base · STS tasks (Avg.) | Spearman's correlation: 78.72$_\mathrm{\pm 0.24}$ -> 79.14$_\mathrm{\pm 0.27}$ | Helped |
| use LLM-generated complex patterns rather than simple data-augmentation patterns | U-CSE · STS tasks | Spearman's correlation | Helped |
| change the anchor-sentence data domain while keeping the similarity pattern fixed | CSE · STS tasks | Spearman's correlation | No clear effect |
Surprising findings
- Data domain (Wiki vs NLI premises) has little effect on STS performance; the similarity pattern is the dominant factor.
- Generating only 20,000 LLM instances is enough to beat data-generation baselines such as CLAIF, which use far more generated data.
Limitations
- RFD conclusions are drawn from a limited number of settings (seven points in two data domains, plus four to six additional settings).
- Experiments are primarily on BERT-base and RoBERTa-base; the LLM used is a single closed-source model (gpt-3.5-turbo-0613).
Experimental setup
Datasets: Wikipedia sentences (unsupervised CSE) NLI dataset (SNLI/MNLI premises and entailment/contradiction hypotheses) STS12 training split (pattern source) STS Benchmark validation split (hyperparameter selection) LLM-generated data from gpt-3.5-turbo-0613
Benchmarks: STS12 STS13 STS14 STS15 STS16 STS Benchmark SICK-R SentEval transfer tasks (MR, CR, SUBJ, MPQA, SST, TREC, MRPC)
Metrics: Spearman's correlation alignment uniformity RFD accuracy
Baselines: SimCSE PromptBERT ConSERT InfoCSE BERT-whitening Dino CLAIF SBERT InferSent
Builds on: SimCSE PromptBERT CLAIF InfoNCE
Closest papers by meaning
- Scalable Attentive Sentence Pair Modeling via Distilled Sentence Embedding (AAAI 2020 · similarity 0.90)
- TNCSE: Tensor Norm Constraints for Unsupervised Contrastive Learning of Sentence Embeddings (AAAI 2025 · similarity 0.90)
- An efficient framework for learning sentence representations (ICLR 2018 · similarity 0.89)
- A Simple but Tough-to-Beat Baseline for Sentence Embeddings (ICLR 2017 · similarity 0.89)
- DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning (AAAI 2024 · similarity 0.88)
- Analyzing Similarity Metrics for Data Selection for Language Model Pretraining (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.