Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning

TNCSE: Tensor Norm Constraints for Unsupervised Contrastive Learning of Sentence Embeddings

AAAI 2025 · Label-efficient and robust learning › Zero-/few-shot and meta-learning · approach: Zero-shot robustness and transfer · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TNCSE, an unsupervised contrastive sentence-embedding framework that adds tensor-norm constraints between positive pairs to a two-encoder (BERT and RoBERTa, RTT-augmented) ensemble, reporting new SOTA on seven STS tasks.

Problem

Unsupervised contrastive sentence embedding methods (e.g., SimCSE) use cosine similarity with InfoNCE, which constrains only direction and ignores the norm (module length) of sentence embeddings.

Approach

A tensor-norm loss L_TN = ||h - h+|| / (||h|| + ||h+||) between positive pairs, applied to pooler outputs and cross-encoder (interaction) pairs, combined with InfoNCE losses in a two-encoder ensemble so the model aligns both angle and norm without relying on other checkpoints.

Key results

• TNCSE achieves 79.58 average on seven STS tasks with BERT-base, above EDFSE (79.04) and RankCSE-based comparisons.
• TNCSE+UC reaches 80.56 (BERT-base) and 80.41 (RoBERTa-base) average on STS, the best among compared models under the unified InfoCSE-ensemble setting.
• On MTEB subsets, TNCSE gets 22.10 retrieval MAP@10 versus 18.98 for SimCSE and 39.34 STS22 accuracy versus 32.78 for SimCSE.

Measured effects reported in the paper

Surprising findings

• Applying the norm loss directly to last hidden states is ineffective because BERT-like last hidden state norms converge to a near-uniform value; removing LayerNorms to restore norm information degrades performance.
• The authors found that reproducing SimCSE with official code and default hyperparameters gave about 74% versus the reported 76.25%, requiring added unlabelled SICKR data to match baselines.

Limitations

• Gains over strong baselines are small (for example 79.58 vs 79.04 for EDFSE), and some reproduced baselines fell short of reported numbers.
• Norm constraint is ineffective on last hidden states of BERT-like models due to LayerNorm; it must be applied to the pooler output, which is a design choice with limited theoretical grounding.
• Evaluation on MTEB uses only a random subset of tasks (30 classification, 30 retrieval) due to compute constraints.

Experimental setup

Datasets: Wiki1M (princeton-nlp datasets-for-simcse) SICKR unlabelled round-trip translation (Google Translate) augmented data

Benchmarks: STS12 STS13 STS14 STS15 STS16 STS-B SICK-R STS17 STS22 MTEB classification MTEB reranking MTEB retrieval

Metrics: Spearman correlation accuracy MAP MAP@10

Baselines: SimCSE DiffCSE ESimCSE ArcCSE InfoCSE PromptBERT SNCSE WhitenedCSE EDFSE RankCSE IS-CSE

Builds on: SimCSE EDFSE InfoNCE RankCSE

Closest papers by meaning

• Instance Smoothed Contrastive Learning for Unsupervised Sentence Embedding (AAAI 2023 · similarity 0.90)
• Narrowing the Gap between Supervised and Unsupervised Sentence Representation Learning with Large Language Model (AAAI 2024 · similarity 0.90)
• Unsupervised Sentence Representation via Contrastive Learning with Mixing Negatives (AAAI 2022 · similarity 0.88)
• DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning (AAAI 2024 · similarity 0.86)
• Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe (NeurIPS 2024 · similarity 0.85)
• SetCSE: Set Operations using Contrastive Learning of Sentence Embeddings (ICLR 2024 · similarity 0.85)