Research map / Label-efficient and robust learning / Knowledge distillation

Implicit Word Reordering with Knowledge Distillation for Cross-Lingual Dependency Parsing

AAAI 2025 · Label-efficient and robust learning › Knowledge distillation · approach: Distillation for machine translation · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes IWR-KD, which distills target-language word order knowledge from a POS-based word-reordering teacher into a biaffine dependency parser student, enabling implicit reordering without generating permuted sentences.

Problem

Word order differences between source and target languages hinder cross-lingual dependency parsing; order-agnostic models lose grammatical order information and explicit word reordering is computationally expensive and can add unnatural noise.

Approach

Instead of permuting source sentences, a teacher trained on target-language POS tag pairs predicts head-dependent word order, and a parser student mimics the teacher's order distributions via MSE distillation alongside its parsing loss, implicitly adapting word order in feature space.

Key results

• IWR-KD achieves average 74.1 UAS / 61.7 LAS across 31 languages, versus 73.4/61.5-range for WOL (73.4/60.8) and 73.0/61.3 for Subtree-EWR, and 72.9/60.5 for mBERT-Direct.
• Ablation: replacing soft order distributions with pseudo-labels drops average to 73.5/61.3; removing fine-tuning drops to 72.6/60.3.
• Word order distance correlates with transfer performance at Pearson -0.8803 (p=1e-10).

Measured effects reported in the paper

Surprising findings

• Explicit word reordering (Subtree-EWR) declines below mBERT in many languages such as sl and fi, attributed to unnatural reordered sentences acting as noise.
• Without fine-tuning mBERT, performance drops by 1.5 UAS, indicating encoder adaptation matters.

Limitations

• Relies on gold POS tags for target languages, an idealized setting; silver POS tags reduce gains.
• Teacher head predictions come from a source-language parser, and the method does not reduce word order distance for some languages such as Chinese.

Experimental setup

Datasets: Universal Dependencies v2.14 treebanks (English source)

Benchmarks: Universal Dependencies v2.14 treebanks (31 languages)

Metrics: UAS LAS

Baselines: SelfAttn-Direct mBERT-Direct Frozen PE Subtree-EWR WOL

Builds on: biaffine dependency parser mBERT Subtree-EWR Frozen position embedding

Closest papers by meaning

• Dynamic Syntactic Feature Filtering and Injecting Networks for Cross-lingual Dependency Parsing (AAAI 2025 · similarity 0.86)
• Order-Independence Without Fine Tuning (NeurIPS 2024 · similarity 0.86)
• WASD: Wasserstein-based Knowledge Distillation for Large Language Models (NeurIPS 2026 · similarity 0.85)
• DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.85)
• Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models (AAAI 2025 · similarity 0.84)
• TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models (ICLR 2025 · similarity 0.84)