Research map / Architectures and efficiency / RNNs and normalization
LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LRC-BERT instead of TinyBERT | BERT-base distillation · GLUE average | average score: 76.5 -> 77.5 | Helped |
| Add COS-NCE transformer-layer distillation (ablation) | LRC-BERT · CoLA dev | Matthew's correlation: 37.0 -> 50.0 | Helped |
| Add COS-NCE transformer-layer distillation (ablation) | LRC-BERT · MNLI-m dev | accuracy: 78.0 -> 83.4 | Helped |
| Add soft prediction loss (ablation) | LRC-BERT · GLUE ablation tasks | task score: 82.7 -> 83.4; 83.0 -> 83.4; 89.4 -> 89.0; 48.8 -> 50.0 | Mixed |
| Add hard prediction loss (ablation) | LRC-BERT · GLUE ablation tasks | task score: 83.0 -> 83.4; 83.5 -> 83.4; 88.7 -> 89.0; 48.6 -> 50.0 | Mixed |
| Use two-stage training, first fitting intermediate-layer contrastive features (ablation) | LRC-BERT · MNLI-m dev | accuracy: 79.4 -> 83.4 | Helped |
| Add gradient perturbation during training (ablation) | LRC-BERT · MNLI-m training | training-loss stability | Helped |
| Add pre-training and data augmentation before task distillation | LRC-BERT_1 · MRPC | task score: +4.0% | Helped |
| Add pre-training and data augmentation before task distillation | LRC-BERT_1 · RTE | task score: +12.1% | Helped |
| Add pre-training and data augmentation before task distillation | LRC-BERT_1 · CoLA | task score: +14.9% | Helped |
| Use angular COS-NCE rather than MSE for intermediate-layer distillation | LRC-BERT · GLUE average | average score: 77.0 -> 77.5 | Helped |
Closest papers by meaning
- Learning to Augment for Data-scarce Domain BERT Knowledge Distillation (AAAI 2021 · similarity 0.89)
- A Compact Model for Mathematics Problem Representations Distilled from BERT (AAAI 2025 · similarity 0.85)
- LLaVA-MoD: Making LLaVA Tiny via MoE-Knowledge Distillation (ICLR 2025 · similarity 0.85)
- WASD: Wasserstein-based Knowledge Distillation for Large Language Models (NeurIPS 2026 · similarity 0.84)
- DDK: Distilling Domain Knowledge for Efficient Large Language Models (NeurIPS 2024 · similarity 0.84)
- A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.