Research map / Vision-language and multimodal / Human and embodied modeling
SetCSE: Set Operations using Contrastive Learning of Sentence Embeddings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SetCSE, a retrieval framework that represents complex semantics as sets of example sentences, fine-tunes sentence embedders with an inter-set contrastive loss, and queries sentences via set intersection/difference operations.
Problem
Single-sentence queries struggle to express complex or intricate semantics that are naturally conveyed collectively by multiple example sentences, limiting sentence retrieval with convoluted prompts.
Approach
Represent a semantic as a set of sentences, fine-tune sentence embedding models with an inter-set contrastive objective that treats sentences from different sets as negatives, and define SetCSE intersection and difference as ranking operations based on average cosine similarity to sets.
Key results
- SetCSE intersection improves accuracy and F1 on average by 39% and 37% over existing models across five datasets (per table caption).
- SetCSE difference improves accuracy and F1 by 14% and 21% on average (per table caption).
- Series of two SetCSE intersections show an average 27% accuracy and 25% F1 improvement over existing models (per caption).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply inter-set contrastive fine-tuning before intersection retrieval | SetCSE intersection · AGT, AGD, FPB, Banking77, FMTOD | accuracy: 39% | Helped |
| Apply inter-set contrastive fine-tuning before difference retrieval | SetCSE difference · AGT, AGD, FPB, Banking77, FMTOD | accuracy: 14% | Helped |
| Use a set of example sentences instead of a single sentence to represent semantics (ablation) | SetCSE querying · Evaluation datasets | accuracy | Helped |
| Set temperature τ to 0.001 instead of 0.05 (ablation) | SetCSE intersection on SimCSE-BERT · AGT | accuracy: 78.29 -> 77.14 | Hurt |
| Set temperature τ to 0.01 instead of 0.05 (ablation) | SetCSE intersection on SimCSE-BERT · AGT | accuracy: 78.29 -> 77.31 | Hurt |
| Set temperature τ to 0.1 instead of 0.05 (ablation) | SetCSE intersection on SimCSE-BERT · AGT | accuracy: 78.29 -> 77.42 | Hurt |
| Set temperature τ to 1 instead of 0.05 (ablation) | SetCSE intersection on SimCSE-BERT · AGT | accuracy: 78.29 -> 77.54 | Hurt |
| Train for 20 rather than 60 epochs (ablation) | SetCSE intersection on SimCSE-BERT · AGT | accuracy: 78.27 -> 71.40 | Hurt |
| Train for 30 rather than 60 epochs (ablation) | SetCSE intersection on SimCSE-BERT · AGT | accuracy: 78.27 -> 74.63 | Hurt |
| Train for 40 rather than 60 epochs (ablation) | SetCSE intersection on SimCSE-BERT · AGT | accuracy: 78.27 -> 75.76 | Hurt |
| Train for 50 rather than 60 epochs (ablation) | SetCSE intersection on SimCSE-BERT · AGT | accuracy: 78.27 -> 75.82 | Hurt |
| Train for 70 rather than 60 epochs (ablation) | SetCSE intersection on SimCSE-BERT · AGT | accuracy: 78.27 -> 79.25 | Helped |
Surprising findings
- Decoder-only SGPT-125M benefits less from inter-set contrastive fine-tuning than encoder models.
- SetCSE intersection performs on par with supervised classification while also enabling complex queries that supervised learning cannot.
Limitations
- Negative pairs grow quadratically with number of sets, requiring random subsampling of pairs.
- SetCSE operations are not strictly commutative and lack some set-theoretic properties.
- Evaluation limited to small embedding models; larger models and standard IR benchmarks left unexplored.
Experimental setup
Datasets: AG News Financial PhraseBank Banking77 FMTOD GitHub Issue English Quotes Reuters-21578 Law Stack Exchange Twitter Stance Evaluation S&P 500 earnings calls
Benchmarks: AG News Financial PhraseBank Banking77 FMTOD GitHub Issue English Quotes Reuters-21578
Metrics: accuracy F1
Baselines: BM25 DPR TFIDF BERT RoBERTa Contriever SGPT-125M SimCSE DiffCSE MCSE
Builds on: SimCSE DiffCSE MCSE
Closest papers by meaning
- TNCSE: Tensor Norm Constraints for Unsupervised Contrastive Learning of Sentence Embeddings (AAAI 2025 · similarity 0.85)
- Narrowing the Gap between Supervised and Unsupervised Sentence Representation Learning with Large Language Model (AAAI 2024 · similarity 0.84)
- A General Framework for Producing Interpretable Semantic Text Embeddings (ICLR 2025 · similarity 0.83)
- Some Like It Small: Czech Semantic Embedding Models for Industry Applications (AAAI 2024 · similarity 0.83)
- Contrastive Retrieval Heads for Improved Attention-Based Reranking (NeurIPS 2026 · similarity 0.83)
- The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentation (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.