Research map / Architectures and efficiency / Attention and transformers
Headless Language Models: Learning without Predicting with Contrastive Weight Tying
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Contrastive Weight Tying (CWT), replacing the cross-entropy/vocabulary projection in LM pretraining with an in-batch contrastive loss that reconstructs input embeddings, yielding Headless LMs that are cheaper and often better on GLUE, LAMBADA, and XNLI.
Problem
Self-supervised language model pretraining relies on predicting distributions over large vocabularies via a projection head, which costs memory and compute and limits scaling to large vocabularies.
Approach
Drop the softmax/vocabulary projection and instead train the model's output representations to contrastively match the input embeddings of the original tokens, using in-batch samples as negatives, as the sole pretraining objective (Contrastive Weight Tying).
Key results
- Headless MLM at equal GPU hours (444 vs 989) reaches 83.29 GLUE average vs 82.43 for vanilla MLM; at 8.2B tokens it reaches 84.08.
- Headless decoder with fine-tuning reaches 22.2% LAMBADA accuracy vs 19.52% for the vanilla causal LM (+2.7).
- Multilingual headless MLM improves XNLI average by 2 points (cross-lingual) and 2.7 points (target-language fine-tuning) over vanilla.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Contrastive Weight Tying instead of vanilla MLM pretraining | BERT-base · GLUE dev at 4.1B pretraining tokens | average score: 82.43 -> 83.29 | Helped |
| Use Contrastive Weight Tying instead of vanilla MLM pretraining | BERT-base · GLUE dev at similar pretraining duration | average score: 82.43 -> 84.08 | Helped |
| Use Contrastive Weight Tying instead of vanilla MLM pretraining | BERT-base · SuperGLUE dev | average accuracy: 67.9 -> 68.6 | Helped |
| Add predictive CLM fine-tuning after CWT pretraining (ablation) | Pythia-70M HLM · LAMBADA | accuracy: 18.26 -> 22.2 | Helped |
| Add predictive CLM fine-tuning after CWT pretraining (ablation) | Pythia-70M HLM · LAMBADA | perplexity: 524.44 -> 153.5 | Helped |
| Use Contrastive Weight Tying instead of vanilla MLM pretraining | DistilBERT multilingual · XNLI dev, English-only fine-tuning | average accuracy: 55.37 -> 57.36 | Helped |
| Use Contrastive Weight Tying instead of vanilla MLM pretraining | DistilBERT multilingual · XNLI dev, target-language fine-tuning | average accuracy: 62.87 -> 65.54 | Helped |
| Use CWT pretraining and predictive fine-tuning instead of vanilla CLM | Pythia-70M · LM Evaluation Harness zero-shot BoolQ | accuracy: 47.8 -> 53.0 | Helped |
| Use CWT pretraining and predictive fine-tuning instead of vanilla CLM | Pythia-70M · LM Evaluation Harness zero-shot PubMedQA | accuracy: 43.8 -> 47.5 | Helped |
| Use CWT pretraining and predictive fine-tuning instead of vanilla CLM | Pythia-70M · LM Evaluation Harness zero-shot QASPER | accuracy: 41.9 -> 66.0 | Helped |
| Use regular rather than balanced cross-entropy for fine-tuning (ablation) | Monolingual encoder models · GLUE dev | average score: 81.5 vs 82.43; 83.19 vs 83.29 | No clear effect |
| Increase the CWT micro-batch size (ablation) | Headless models · LAMBADA | accuracy | Helped |
Surprising findings
- The raw headless causal model without predictive fine-tuning performs poorly on LAMBADA (524 perplexity), so a short CLM fine-tuning stage was needed to recover generation ability.
- Headless MLM outperforms its vanilla counterpart on GLUE only with balanced cross-entropy fine-tuning; with regular cross-entropy the gap is smaller on some tasks (e.g., MRPC favors vanilla).
Limitations
- Experiments are limited to small-scale architectures (≈11M–137M parameters), with no evaluation of large language models.
- Headless decoders cannot generate text directly and require an extra predictive fine-tuning stage on a small portion of data.
- Only English monolingual and a single multilingual setting were studied; the method was not compared with ELECTRA, DeBERTa, or other efficient encoder objectives.
Experimental setup
Datasets: OpenWebText2 multilingual Wikipedia CC-News GLUE SuperGLUE LAMBADA XNLI WordNet
Benchmarks: GLUE SuperGLUE LAMBADA LM Evaluation Harness subset (ARC, BoolQ, CrowS-Pairs, RACE, SciQ, PubMedQA, QASPER) XNLI
Metrics: accuracy Matthews correlation Spearman correlation perplexity GPU hours stereotype percentage
Baselines: Vanilla MLM (BERT-base) Vanilla CLM (Pythia-70M) Vanilla multilingual MLM (distilled mBERT)
Builds on: BERT weight tying Pythia ELECTRA SimCTG
Closest papers by meaning
- Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch (ICML 2024 · similarity 0.86)
- Constrained Entropic Unlearning: A Primal-Dual Framework for Large Language Models (NeurIPS 2025 · similarity 0.86)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.85)
- Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models (NeurIPS 2024 · similarity 0.85)
- Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.85)
- A Simple and Effective Pruning Approach for Large Language Models (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.