Research map / Label-efficient and robust learning / Knowledge distillation
Scalable Attentive Sentence Pair Modeling via Distilled Sentence Embedding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace pairwise cross-attention scoring with distilled sentence embeddings and a low-cost similarity function | BERT-Large sentence-pair scoring · GLUE sentence-pair tasks | average score: 86.82 -> 82.83 | Hurt |
| replace pairwise cross-attention scoring with distilled sentence embeddings and a low-cost similarity function | BERT-Large · Offline 1M sentence-pair similarities | computation time: 9.66hr -> 37.21s | Helped |
| replace pairwise cross-attention scoring with distilled sentence embeddings and a low-cost similarity function | BERT-Large · Online 100K query-candidate similarities | computation time: 58m -> 0.256s | Helped |
| add teacher-score distillation (ablation) | DSE · GLUE sentence-pair tasks | average score: 82.2 -> 82.83 | Helped |
| add teacher-score distillation (ablation) | DSE · STS-B | Pearson/Spearman correlation: 87.0/86.5 -> 86.4/85.8 | Hurt |
| add teacher-score distillation (ablation) | DSE universal embeddings · Universal sentence embedding benchmarks | average score: 85.9 -> 86.4 | Helped |
| fine-tune the sentence embedding function instead of freezing it (ablation) | DSE (α = 1) · GLUE sentence-pair tasks | average score: 74.96 -> 82.76 | Helped |
| use α = 0.5 rather than α = 1 for the distillation/label-loss balance (ablation) | DSE · GLUE sentence-pair tasks | average score: 82.76 -> 82.83 | No clear effect |
Closest papers by meaning
- Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations (ICLR 2022 · similarity 0.91)
- Narrowing the Gap between Supervised and Unsupervised Sentence Representation Learning with Large Language Model (AAAI 2024 · similarity 0.90)
- A Simple but Tough-to-Beat Baseline for Sentence Embeddings (ICLR 2017 · similarity 0.90)
- Instance Smoothed Contrastive Learning for Unsupervised Sentence Embedding (AAAI 2023 · similarity 0.89)
- TNCSE: Tensor Norm Constraints for Unsupervised Contrastive Learning of Sentence Embeddings (AAAI 2025 · similarity 0.87)
- Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.