Research map / Label-efficient and robust learning / Knowledge distillation
Ensemble Distillation for Unsupervised Constituency Parsing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes tree averaging via a CYK-like MBR decoding that combines seven unsupervised constituency parsers into an ensemble, then distills it into an RNNG student to reach state-of-the-art unsupervised parsing on PTB and domain-shift SUSANNE.
Problem
Unsupervised constituency parsing has relied on heuristics or indirect supervision, and existing parsers with similar F1 make very different errors, so their expertise is not exploited; multi-teacher distillation also suffers from over-smoothing when teachers are heterogeneous.
Approach
Define an average tree as the binary tree maximizing total constituent hit count across teachers (equivalently total F1 against teachers), solved exactly with a CYK-like dynamic program over the full tree space (generative MBR), then distill that single ensemble tree into an RNNG student (followed by URNNG refinement) instead of union distillation.
Key results
- Ensemble of seven teachers reaches 70.4 mean F1 on PTB without distillation, and the RNNG+URNNG student reaches 72.8 F1, a new state of the art over the best single teacher at 62.9 mean.
- On PTB-to-SUSANNE domain shift the ensemble reaches 50.3 F1, outperforming selective MBR (47.4) by about 3 points.
- Union distillation gives 65.6 F1 versus 71.1 for the ensemble-then-distill RNNG, and its prediction entropy is 11.42 versus 4.93.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the best single teacher with the corresponding-run ensemble | Unsupervised constituency parsing on PTB · PTB test | F1: 62.9 -> 70.4 | Helped |
| Search the full binary-tree space with generative MBR rather than selecting among teacher outputs | Our ensemble · PTB test | F1: 66.3 -> 70.4 | Helped |
| Use the best teacher from each model across runs instead of corresponding runs | Our ensemble · PTB test | F1: 70.4 -> 71.9 | Helped |
| Use the worst teacher from each model across runs instead of corresponding runs | Our ensemble · PTB test | F1: 70.4 -> 69.4 | Hurt |
| Add RNNG distillation (ablation) | Our ensemble (corresponding run) · PTB test | F1: 70.5 -> 69.7 | Hurt |
| Add URNNG refinement after RNNG distillation (ablation) | Our ensemble (corresponding run) · PTB test | F1: 69.7 -> 71.7 | Helped |
| Add URNNG refinement after RNNG distillation (ablation) | Our ensemble (best teachers) · PTB test | F1: 71.1 -> 72.8 | Helped |
| Train the student from the union of teacher outputs rather than distilling the ensemble's single best tree | Student parser · PTB test | F1: 65.6 vs. 68.6 | Hurt |
| Add RNNG distillation to the ensemble (ablation) | Our ensemble · SUSANNE test (PTB-trained) | F1: 50.3 -> 49.1 | Hurt |
| Add URNNG refinement after RNNG distillation (ablation) | Our ensemble · SUSANNE test (PTB-trained) | F1: 49.1 -> 48.8 | Hurt |
| Use the generative ensemble instead of selective MBR | Unsupervised constituency parsing under domain shift · SUSANNE test (PTB-trained) | F1: 47.4 -> 50.3 | Helped |
| Distill ensemble knowledge into an RNNG student instead of using the ensemble at inference (ablation) | Our ensemble · PTB test | inference time: 7,541 -> 410 | Helped |
Surprising findings
- Correlation between different unsupervised parsers is lower than between restarts of the same model, suggesting heterogeneous expertise.
- Union distillation performs worse than the best single teacher on PTB.
Limitations
- Student URNNG gains are smaller because the ensemble is already strong
- Teachers are trained with unlabeled data only but rely on a labeled dev set for checkpoint selection
- Evaluation limited to English PTB and SUSANNE; multilingual settings are left as future work
Experimental setup
Datasets: Penn Treebank SUSANNE
Benchmarks: Penn Treebank SUSANNE
Metrics: unlabeled F1 recall by constituent label inference time prediction entropy
Baselines: Ordered Neurons Neural PCFG Compound PCFG DIORA S-DIORA ConTest ContexDistort Selective MBR Union distillation
Builds on: Compound PCFG DIORA URNNG Minimum Bayes risk decoding knowledge distillation
Closest papers by meaning
- Improving Unsupervised Constituency Parsing via Maximizing Semantic Information (ICLR 2025 · similarity 0.89)
- Error Diversity Matters: An Error-Resistant Ensemble Method for Unsupervised Dependency Parsing (AAAI 2025 · similarity 0.88)
- Improving Sequence-to-Sequence Constituency Parsing (AAAI 2018 · similarity 0.87)
- Deep Clustering of Text Representations for Supervision-Free Probing of Syntax (AAAI 2022 · similarity 0.86)
- Ensembling Language Models with Sequential Monte Carlo (NeurIPS 2026 · similarity 0.85)
- Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.