Research map / Label-efficient and robust learning / Recognition and long-tail
Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Alternate bi-encoder and cross-encoder training with pseudo-labels | SimCSE-based sentence encoders · English STS (average) | Spearman's ρ: 76.21 -> 78.65 | Helped |
| Add mutual-distillation between parallel PLMs | Trans-Encoder · English STS (average) | Spearman's ρ: 78.65 -> 80.41 | Helped |
| Alternate bi-encoder and cross-encoder training with mutual-distillation | SimCSE-BERT-base · QQP | AUC: 80.38 -> 84.29 | Helped |
| Alternate bi-encoder and cross-encoder training with mutual-distillation | SimCSE-BERT-base · QNLI | AUC: 71.38 -> 77.11 | Helped |
| Initialize stages from original PLM weights rather than pass weights sequentially (ablation) | Trans-Encoder · English STS (average) | Spearman's ρ: 78.08 -> 78.65 | Helped |
| Use one bi-encoder formulation throughout instead of alternating bi- and cross-encoder formulations (ablation) | SimCSE-BERT-base · English STS (average) | Spearman's ρ: 77.16 vs. 78.65 | Hurt |
| Use one bi-encoder formulation throughout instead of alternating bi- and cross-encoder formulations (ablation) | SimCSE-BERT-base · QNLI | AUC: 70.95 vs. 75.30 | Hurt |
| Use MSE instead of BCE for bi-to-cross distillation (ablation) | Trans-Encoder · Bi-to-cross distillation | Convergence/generalization | Hurt |
| Use BCE instead of MSE for cross-to-bi distillation (ablation) | Trans-Encoder · Cross-to-bi distillation | Convergence | Hurt |
| Use Log-Cosh instead of MSE for cross-to-bi regression (ablation) | Trans-Encoder · Cross-to-bi distillation | Performance | No clear effect |
| Train on STS data and transfer to binary-classification tasks | Trans-Encoder (bi) · QQP, QNLI, and MRPC | AUC | Mixed |
| Train on STS data and transfer to binary-classification tasks | Trans-Encoder (cross) · QQP, QNLI, and MRPC | AUC | Helped |
Closest papers by meaning
- Scalable Attentive Sentence Pair Modeling via Distilled Sentence Embedding (AAAI 2020 · similarity 0.91)
- Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (ICLR 2018 · similarity 0.89)
- Narrowing the Gap between Supervised and Unsupervised Sentence Representation Learning with Large Language Model (AAAI 2024 · similarity 0.89)
- Poly-encoders: Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring (ICLR 2020 · similarity 0.89)
- Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings (NeurIPS 2022 · similarity 0.88)
- Siamese Recurrent Architectures for Learning Sentence Similarity (AAAI 2016 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.