Research map / Architectures and efficiency / Pruning and efficient tuning
ShareBERT: Embeddings Are Capable of Learning Hidden Layers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes embedding parameter-sharing (EPS and virtual EPS) that generates transformer encoder weights from embedding columns, yielding ShareBERT which keeps 95.5% of BERT Base GLUE performance with 5M parameters.
Problem
Pre-trained language models are too large for memory-limited devices, and small models usually lose substantial accuracy without knowledge distillation; input embeddings are often ignored in compression.
Approach
Share embedding matrix columns with the hidden-layer weights of a single shared encoder layer (EPS), and use a linear projection to build a virtual embedding matrix (VEPS) when the embedding dimension is too small, so that encoders need near-zero extra parameters.
Key results
- ShareBERT Small (5.0M params, 21.9x fewer) reaches 95.5% of BERT Base GLUE dev performance without knowledge distillation.
- ShareBERT Base (13.8M) reaches 97.4% and Large (27.0M, 6 layers) matches BERT Base at 100% of GLUE dev average.
- Virtual EPS improves word similarity average correlation by up to 0.153 and word analogy accuracy by up to 5.3% over BERT Base.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply EPS to BERT Base (ablation) | BERT Base · GLUE dev | average score: 81.4 -> 77.0 | Hurt |
| Increase feed-forward size from I=3072 to I=6144 (ablation) | BERT + EPS · GLUE dev | average score: 77.0 -> 76.5 | Hurt |
| Add embedding factorization and increase hidden size (ablation) | BERT + EPS · GLUE dev | average score: 77.0 -> 80.9 | Helped |
| Use virtual EPS with F=128 and V=2048 (ablation) | ShareBERT · GLUE dev | average score: 77.0 -> 77.8 | Helped |
| Increase embedding factorization dimension from F=128 to F=384 (ablation) | ShareBERT · GLUE dev | average score: 77.8 -> 79.3 | Helped |
| Increase embedding factorization dimension from F=384 to F=768 and use L=6 (ablation) | ShareBERT · GLUE dev | average score: 79.3 -> 81.4 | Helped |
| Apply VEPS instead of vanilla EPS (ablation) | BERT embeddings · Word Similarity average | Spearman correlation: 0.485 -> 0.594 | Helped |
| Apply VEPS instead of vanilla EPS (ablation) | BERT embeddings · Word Analogy | accuracy: 9.9% -> 10.7% | Helped |
| Apply vanilla EPS (ablation) | BERT embeddings · Word Analogy | accuracy: 9.9% -> 3.3% | Hurt |
| Apply EPS to shared linear transformations (ablation) | Simple-Share MT · English-Vietnamese NMT | BLEU: 29.85 -> 29.58 | Hurt |
| Apply EPS to shared linear transformations (ablation) | Simple-Share IC · Image Captioning | CIDEr-D: 130.8 -> 132.6 | Helped |
Surprising findings
- Virtual embedding parameter-sharing improved word-representation quality (word similarity and analogy) despite the shared parameters.
- ShareBERT variants trained with near-zero encoder parameters match or exceed BERT probing F1 at embedding-only probing depth.
Limitations
- Reduces parameters but increases computation (about 4.5x more FLOPs than BERT for Small/Base).
- Vanilla EPS degrades GLUE by about 4.4% and harms word analogy; VEPS needed to recover performance.
- Pretraining and hardware details (compute) are not reported; code promised but not yet linked.
Experimental setup
Datasets: English Wikipedia BookCorpus GLUE Word Analogy WordSim-353 RW SCWS edge probing tasks
Benchmarks: GLUE Word Analogy Word Similarity edge probing WMT En-Vi NMT image captioning COCO
Metrics: accuracy F1 Spearman correlation BLEU CIDEr-D relative performance vs BERT Base
Baselines: BERT Base TinyBERT4 ALBERT Base DistilBERT MobileBERT BERT-of-Theseus Simple-Share
Builds on: BERT ALBERT Transformer Knowledge Distillation
Closest papers by meaning
- Layer by Layer: Uncovering Hidden Representations in Language Models (ICML 2025 · similarity 0.86)
- Language Model Memory and Memory Models for Language (NeurIPS 2026 · similarity 0.86)
- Scaling Embedding Layers in Language Models (NeurIPS 2025 · similarity 0.85)
- Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression (ICLR 2025 · similarity 0.85)
- Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models (NeurIPS 2024 · similarity 0.84)
- Your Mixture-of-Experts LLM Is Secretly an Embedding Model for Free (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.