Research map / Label-efficient and robust learning / Knowledge distillation
Progressive Distillation Based on Masked Generation Feature Method for Knowledge Graph Completion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PMD, a progressive distillation method for description-based knowledge graph completion that combines masked generation feature distillation with grade-by-grade compression of a BERT-based SimKGC model.
Problem
Description-based knowledge graph completion models built on pre-trained language models have too many parameters and high computational cost for practical deployment, and no distillation method existed for this family.
Approach
Masked generation feature distillation (MGFD) aligns student and teacher features at masked token positions to transfer richer representations than traditional feature distillation, combined with a progressive scheme that lowers mask ratio and layer count across student grades (12, 9, 6, 3 layers).
Key results
- PMD12 (210M, 12 layers) reaches MRR 67.8 and Hits@10 83.2 on WN18RR, surpassing the baseline's 67.1 MRR and 81.7 Hits@10.
- The smallest student PMD3 (91M parameters) reaches MRR 62.8 on WN18RR, a 56.7% parameter reduction from the 210M baseline.
- On FB15K-237, PMD12 improves Hits@3 and Hits@10 but decreases Hits@1 relative to the baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply PMD pre-distillation with MGFD | SimKGC baseline · WN18RR | MRR: 67.1 -> 67.8 | Helped |
| Apply PMD pre-distillation with MGFD | SimKGC baseline · WN18RR | Hits@10: 81.7 -> 83.2 | Helped |
| Apply PMD pre-distillation with MGFD | SimKGC baseline · FB15K-237 | MRR: 33.3 -> 33.3 | No clear effect |
| Apply PMD pre-distillation with MGFD | SimKGC baseline · FB15K-237 | Hits@1: 24.6 -> 24.3 | Hurt |
| Compress the model to 3 Transformer layers with progressive distillation | SimKGC baseline · WN18RR | Parameters: 56.7% | Helped |
| Remove MGFD (ablation) | PMD · WN18RR, 12 layers | Hits@1: 58.8 -> 59.3 | Helped |
| Remove MGFD (ablation) | PMD · WN18RR, 12 layers | Hits@3: 73.7 -> 72.5 | Hurt |
| Remove MGFD (ablation) | PMD · WN18RR, 12 layers | Hits@10: 83.2 -> 81.3 | Hurt |
| Replace MGFD with logits distillation (ablation) | PMD · WN18RR, 12 layers | MRR: 67.8 -> 67.1 | Hurt |
| Replace MGFD with layer-skipping feature distillation (ablation) | PMD · WN18RR, 12 layers | MRR: 67.8 -> 67.0 | Hurt |
| Increase MGFD masking rate from 0% to 50% (ablation) | PMD12 pre-distillation · WN18RR | Hits@10: 81.67 to 84.27 and then decreasing to 83.75 | Mixed |
| Increase MGFD masking rate from 0% to 50% (ablation) | PMD12 pre-distillation · WN18RR | Hits@1: 58.44 to 53 | Hurt |
Surprising findings
- Removing MGFD and using only progressive distillation improves Hits@1 but lowers Hits@3 and Hits@10 relative to the full PMD.
- On FB15K-237, PMD12 drops Hits@1 compared to the baseline, which the authors attribute to teacher errors caused by many relations per entity.
Limitations
- Gains on FB15K-237 are mixed, with Hits@1 decreasing for the 12-layer model.
- Mask-rate and loss weights are tuned by grid search, and the authors acknowledge that adaptive mask selection is left for future work.
- Baseline numbers for other methods are taken from original papers rather than re-run under a common setup.
Experimental setup
Datasets: WN18RR FB15K-237
Benchmarks: WN18RR FB15K-237
Metrics: MRR Hits@1 Hits@3 Hits@10 MR
Baselines: TransE RotatE ConvE CompGCN KG-BERT MTL-KGC C-LMKE KGLM LP-BERT StAR SimKGC LKD
Builds on: SimKGC BERT knowledge distillation PKD masked language modeling
Closest papers by meaning
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.89)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.87)
- DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.87)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.87)
- DDK: Distilling Domain Knowledge for Efficient Large Language Models (NeurIPS 2024 · similarity 0.87)
- DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.