Research map / Label-efficient and robust learning / Knowledge distillation
Discrepancy and Uncertainty Aware Denoising Knowledge Distillation for Zero-Shot Cross-Lingual Named Entity Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DenKD, a knowledge-distillation model for zero-shot cross-lingual NER that denoises pseudo-labels via discrepancy-aware adversarial representation learning for the teacher and uncertainty-weighted distillation using Monte-Carlo dropout variance.
Problem
Pseudo-label noise caused by the source-target language gap in knowledge-distillation-based zero-shot cross-lingual NER, which can cause negative knowledge transfer to the student network.
Approach
Uses a two-classifier teacher trained adversarially (maximum classifier discrepancy with a gradient reversal layer) to create more discriminative target representations and better pseudo-labels, and weights the student's distillation loss by Monte-Carlo dropout prediction variance as a pseudo-label uncertainty estimate.
Key results
- DenKD reaches 73.92 average F1 across 28 WikiANN target languages versus 71.45 for ProKD (about +2.47 points).
- Ablation: removing discrepancy-aware representation learning drops average F1 by 2.91 (to 67.21).
- Monte-Carlo variance uncertainty outperforms two-classifier discrepancy as an uncertainty signal (ablation drop of 0.52 points).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply DenKD instead of ProKD | zero-shot cross-lingual NER · Wikiann 28-language Avg | token-level F1 (%): 71.45 -> 73.92; 2.47% | Helped |
| add discrepancy-aware representation learning (ablation) | DenKD teacher network · Wikiann target-language Avg (Table 2) | token-level F1 (%): 67.21 -> 70.12; ↑2.91 | Helped |
| add uncertainty-aware distillation weighting (ablation) | DenKD student network · Wikiann target-language Avg (Table 2) | token-level F1 (%): 69.13 -> 70.12; ↑0.99 | Helped |
| use Monte-Carlo variance instead of two-classifier discrepancy for uncertainty estimation (ablation) | DenKD · Wikiann target-language Avg (Table 2) | token-level F1 (%): 69.60 -> 70.12; ↑0.52 | Helped |
| combine multiple teacher predictions instead of using one inference (ablation) | DenKD · Wikiann target-language Avg (Table 2) | token-level F1 (%): 69.74 -> 70.12; ↑0.38 | Helped |
Surprising findings
- Two-classifier discrepancy essentially fails to track pseudo-label noise after max-min adversarial training, as discrepancies approach 0.
Limitations
- Monte-Carlo inference with M samples adds time cost at pseudo-label generation.
- Teacher-student architecture increases GPU memory consumption relative to direct transfer methods.
- Does not compare against methods using extra resources such as machine translation or large unlabeled corpora (UniTrans, ContProto).
Experimental setup
Datasets: WikiANN
Benchmarks: WikiANN
Metrics: token-level F1
Baselines: FTDT Single-TS RIKD AdvPicker DualNER MSD ProKD
Builds on: Single-TS Maximum Classifier Discrepancy Monte-Carlo dropout ProKD Hinton knowledge distillation
Closest papers by meaning
- DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer (NeurIPS 2025 · similarity 0.86)
- DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models (ICML 2025 · similarity 0.85)
- UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition (ICLR 2024 · similarity 0.85)
- Adaptive Integration of Partial Label Learning and Negative Learning for Enhanced Noisy Label Learning (AAAI 2024 · similarity 0.85)
- MCL-NER: Cross-Lingual Named Entity Recognition via Multi-View Contrastive Learning (AAAI 2024 · similarity 0.85)
- Debiased Distillation for Consistency Regularization (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.