Research map / Vision-language and multimodal / CLIP and zero-shot
LexMAE: Lexicon-Bottlenecked Pretraining for Large-Scale Retrieval
Tao Shen, Xiubo Geng, Chongyang Tao, Can Xu, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain with lexicon-bottlenecked masked autoencoding rather than the strongest baseline initialization | Lexicon-weighting retrieval on MS-Marco Dev passage · MS-Marco Dev passage | MRR@10: 41.1 -> 42.6 | Improved |
| Pretrain with LexMAE rather than coCondenser initialization | Lexicon-weighting retrieval on MS-Marco Dev passage · MS-Marco Dev passage | MRR@10: +5.1% MRR@10 | Improved |
| Use LexMAE for zero-shot transfer | BEIR retrieval · BEIR zero-shot average (12 datasets) | nDCG@10: 47.0 -> 48.7 | Improved |
| Keep the CBoW bottleneck rather than disabling it (ablation) | LexMAE pretraining for MS-Marco Dev passage retrieval · MS-Marco Dev passage, first-stage fine-tuning | MRR@10: 38.5 -> 39.3 | Improved |
| Use a CBoW bottleneck rather than a dense [CLS] bottleneck (ablation) | LexMAE pretraining for MS-Marco Dev passage retrieval · MS-Marco Dev passage, first-stage fine-tuning | MRR@10: 38.6 -> 39.3 | Improved |
| Use softmax normalization rather than saturated norm in the CBoW bottleneck (ablation) | LexMAE pretraining for MS-Marco Dev passage retrieval · MS-Marco Dev passage, first-stage fine-tuning | MRR@10: 39.2 -> 39.3 | Improved |
| Stop gradients from flowing to the bottleneck word embeddings (ablation) | LexMAE pretraining for MS-Marco Dev passage retrieval · MS-Marco Dev passage, first-stage fine-tuning | MRR@10: 38.9 -> 39.3 | Improved |
| Keep encoder and decoder LM heads unshared (ablation) | LexMAE pretraining for MS-Marco Dev passage retrieval · MS-Marco Dev passage, first-stage fine-tuning | MRR@10: 39.1 -> 39.3 | Improved |
| Do not add a separate LM head for bottleneck logits (ablation) | LexMAE pretraining for MS-Marco Dev passage retrieval · MS-Marco Dev passage, first-stage fine-tuning | MRR@10: 39.2 -> 39.3 | Improved |
| Use BERT rather than DistilBERT initialization (ablation) | LexMAE pretraining for MS-Marco Dev passage retrieval · MS-Marco Dev passage, first-stage fine-tuning | MRR@10: 38.4 -> 39.3 | Improved |
| Use the inclusive masking strategy rather than exclusive masking (ablation) | LexMAE pretraining for MS-Marco Dev passage retrieval · MS-Marco Dev passage, first-stage fine-tuning | MRR@10: 39.1 -> 39.3 | Improved |
| Use inclusive rather than fully random masking (ablation) | LexMAE pretraining for MS-Marco Dev passage retrieval · MS-Marco Dev passage, first-stage fine-tuning | MRR@10: 39.2 -> 39.3 | No clear effect |
Later papers that compared with this method
1 later accepted paper reports LexMAE as a baseline in their results tables; 0 of them report a better number on most of the results they share with it.
- Retrieval-based Disentangled Representation Learning with Natural Language Supervision (ICLR 2024 · better on 0 of 1 shared result)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More (ICML 2025 · similarity 0.88)
- Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERT (ICML 2024 · similarity 0.87)
- Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale (NeurIPS 2026 · similarity 0.86)
- Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.85)
- Language Modeling Is Compression (ICLR 2024 · similarity 0.85)
- Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
