Research map / Label-efficient and robust learning / Recognition and long-tail
No Training Required: Exploring Random Encoders for Sentence Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ESN instead of BOREP or RandLSTM | Random sentence encoders · SentEval downstream tasks | performance | Helped |
| Replace 300d bag-of-embeddings with 4096d BOREP | Sentence classification baseline · SentEval downstream tasks | performance | Helped |
| Train InferSent-3 rather than use the best random encoder | Sentence representations · SentEval downstream tasks | average performance: 1.7 | Helped |
| Use ST-LN rather than the best random encoder | Sentence representations · SentEval downstream tasks | average performance difference: -0.4 | Mixed |
| Increase random encoder dimensionality | BOREP, RandLSTM, and ESN · SentEval downstream tasks | performance | Mixed |
| Use random projection to 4096×6 dimensions | InferSent-3 and ST-LN · SentEval downstream tasks | performance | Hurt |
| Use sorted batches rather than unsorted batches when max-pooling over padding | RandLSTM, InferSent, and GenSen · MR, CR, MPQA, SUBJ, SST2 | accuracy | Helped |
| Use ReLU to reproduce the sparsity effect of max-pooling over padded values (ablation) | Random sentence encoders · Single-sentence binary classification tasks | accuracy | No clear effect |
| Change BOREP or RandLSTM initialization from heuristic to orthogonal or Xavier (ablation) | BOREP and RandLSTM · SentEval downstream tasks | average performance | Mixed |
| Change RandLSTM initialization from heuristic to Normal (ablation) | RandLSTM · SentEval downstream tasks | average performance: 81.8 -> 69.4 | Hurt |
| Replace pre-trained word embeddings with random word embeddings (ablation) | BOREP and RandLSTM · SentEval downstream tasks | average performance: 5.4 points | Hurt |
| Add positional encoding to word embeddings (ablation) | BOREP · Sentence classification | performance | No clear effect |
Closest papers by meaning
- A Simple but Tough-to-Beat Baseline for Sentence Embeddings (ICLR 2017 · similarity 0.92)
- Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks (ICLR 2017 · similarity 0.91)
- An efficient framework for learning sentence representations (ICLR 2018 · similarity 0.90)
- Narrowing the Gap between Supervised and Unsupervised Sentence Representation Learning with Large Language Model (AAAI 2024 · similarity 0.89)
- TNCSE: Tensor Norm Constraints for Unsupervised Contrastive Learning of Sentence Embeddings (AAAI 2025 · similarity 0.86)
- Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.