Research map / Architectures and efficiency / Attention and transformers
Poly-encoders: Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a Poly-encoder with candidate-conditioned attention over context codes instead of separate single-vector encoding | Bi-encoder · ConvAI2 test | R@1/20: 84.8 $\pm$ 0.1 -> 86.8 $\pm$ 0.1 | Helped |
| Pre-train on Reddit dialogue data instead of Toronto Books and Wikipedia | Poly-encoder 16 · ConvAI2 test | R@1/20: 81.4 $\pm$ 0.2 -> 86.4 $\pm$ 0.3 | Helped |
| Increase the Poly-encoder context-code count (ablation) | Poly-encoder, Reddit pre-training · ConvAI2 test | R@1/20: 86.3 $\pm$ 0.3 -> 86.5 $\pm$ 0.2 -> 86.8 $\pm$ 0.1 | Helped |
| Increase the number of training negatives (ablation) | BERT Bi-encoder · ConvAI2 validation | R@1/20: 81.0 -> 83.3 | Helped |
| Fine-tune all layers except word embeddings rather than only the top layer (ablation) | BERT Bi-encoder · ConvAI2 validation | R@1/20: 74.2 -> 83.3 | Helped |
| Use the first transformer output rather than average pooling (ablation) | BERT Bi-encoder · ConvAI2 validation | R@1/20: 83.1 -> 83.3 | Helped |
| Pre-train on Reddit dialogue data rather than Toronto Books and Wikipedia | Bi-, Poly-, and Cross-encoders · Dialogue tasks | R@1 | Helped |
| Use Poly-encoder 360 instead of Cross-encoder | Reddit-pre-trained models · ConvAI2 test | R@1/20: 87.9 $\pm$ 0.2 -> 86.8 $\pm$ 0.1 | Hurt |
Closest papers by meaning
- Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations (ICLR 2022 · similarity 0.89)
- How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Bias (ICML 2025 · similarity 0.86)
- On the Power of Convolution-Augmented Transformer (AAAI 2025 · similarity 0.85)
- Progressive Inference: Explaining Decoder-Only Sequence Classification Models Using Intermediate Predictions (ICML 2024 · similarity 0.85)
- Towards Understanding Self-Pretraining for Sequence Classification (NeurIPS 2026 · similarity 0.85)
- Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.