Research map / Label-efficient and robust learning / Self-supervised and contrastive

DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning

AAAI 2024 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Representation evaluation and trade-offs · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DenoSent, a self-supervised sentence encoder trained with an intra-sentence denoising auto-encoder objective (back-translation and heavy dropout noise) that is combined with a SimCSE-style contrastive loss to improve sentence embeddings on STS and transfer tasks.

Problem

Contrastive sentence representation methods learn only from inter-sentence relationships, which limits their ability to capture fine-grained sentence semantics.

Approach

Train a Transformer encoder-decoder where the encoder output is pooled to a single sentence vector that conditions the decoder via cross-attention, to reconstruct the original sentence from a noisy version (back-translated and heavily dropped-out). The denoising loss is added to a contrastive loss so both intra- and inter-sentence signals are used.

Key results

• DenoSent-BERT-base achieves 79.33 average Spearman on 7 STS tasks, the best among compared methods in the paper's table (SNCSE 78.97, PromptBERT 78.54).
• Contrastive-only DenoSent reaches 77.96 average STS, a 1.71-point gain over SimCSE (76.25).
• On classification, DenoSent averages 64.46 accuracy versus 63.78 for PromptBERT and 62.73 for SimCSE.

Measured effects reported in the paper

Surprising findings

• Back-translation discrete noise gave slightly better results than gpt-3.5-turbo rewriting (79.33 vs 78.98 average STS)
• Adding more decoder attention heads reduced performance, since the encoder memory has length 1

Limitations

• Gains over the best contrastive methods are modest and the denoising-only variant trails contrastive-only on STS
• Hyperparameters (dropout rate, decoder depth) were tuned on STS-B dev set, and results depend on the back-translation pipeline
• Relies on the sentence pooled to one vector, which limits the information bottleneck design choices

Experimental setup

Datasets: Wiki (SimCSE unsupervised corpus) SentEval STS tasks MTEB tasks

Benchmarks: STS12 STS13 STS14 STS15 STS16 STS-B SICK-R AskUbuntuDupQuestions MindSmallReranking SciDocsRR StackOverflowDupQuestions QuoraRetrieval

Metrics: Spearman correlation MRR@1 MAP@1 classification accuracy

Baselines: SimCSE ConSERT DCLR DiffCSE PromptBERT SNCSE PaSeR CMLM BERT-flow BERT-whitening

Builds on: SimCSE BERT PaSeR InfoNCE

Closest papers by meaning

• Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings (NeurIPS 2022 · similarity 0.88)
• Instance Smoothed Contrastive Learning for Unsupervised Sentence Embedding (AAAI 2023 · similarity 0.88)
• Narrowing the Gap between Supervised and Unsupervised Sentence Representation Learning with Large Language Model (AAAI 2024 · similarity 0.88)
• InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationales (ICLR 2025 · similarity 0.87)
• TNCSE: Tensor Norm Constraints for Unsupervised Contrastive Learning of Sentence Embeddings (AAAI 2025 · similarity 0.86)
• Denoising Time Matters: Diverse Generation in Diffusion Language Models (NeurIPS 2026 · similarity 0.85)