Research map / Label-efficient and robust learning / Recognition and long-tail
Error Diversity Matters: An Error-Resistant Ensemble Method for Unsupervised Dependency Parsing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds an ensemble of unsupervised dependency parsers via MBR-style post hoc aggregation of parse trees (UAS-based, solved with Eisner-style DP) and selects members with a new "society entropy" diversity metric that captures error diversity.
Problem
Unsupervised dependency parsing ensembles are fragile: adding weak individual parsers causes error accumulation and performance drops, and existing ensemble-selection diversity metrics ignore error diversity.
Approach
Aggregate parses by maximizing summed sentence-level UAS to individuals (reducible to MBR decoding solvable with an O(n^3) projective parsing DP), and select individuals via forward stepwise selection with an objective combining validation UAS and society entropy, a diversity measure over the distribution of predicted heads that accounts for both expertise and error diversity.
Key results
- Selected ensemble with society entropy reaches 68.8 UAS on WSJ test, above the best individual Sib&L-NDMV (67.9) and the unselected weighted ensemble (68.4).
- Without selection, the unweighted ensemble of five-run individuals reaches 66.6 weighted UAS, outperforming the best single run (64.3) and CIM aggregation (58.8).
- Society entropy selection achieves ensemble UAS comparable to forward-stepwise ensemble validation (67.1 vs 67.0 overall) while being roughly 500x faster.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UAS-based MBR aggregation instead of CIM aggregation | Unsupervised dependency parsing on WSJ · WSJ test set | UAS: 58.8_{\pm0.7} -> {{65.7}}_{\pm0.9}^{**} | Helped |
| Weight individuals by validation performance (ablation) | Our parse aggregation · WSJ test set | UAS: {{65.7}}_{\pm0.9}^{**} -> {{66.6}}_{\pm0.4}^{***} | Helped |
| Select individuals using society entropy rather than selecting without diversity (ablation) | Our aggregation · WSJ test set | UAS: 63.8 -> \underline{67.3} | Helped |
| Select individuals using society entropy rather than Kuncheva's diversity | Our aggregation · Overall, selected 3, 4, or 5 individuals from 15 | UAS: 66.3_{\pm1.2} -> \bf{67.1_{\pm0.4}} | Helped |
| Use society entropy rather than no diversity during selection (ablation) | Ensemble selection · Overall, selected 3, 4, or 5 individuals from 15 | UAS: 65.2_{\pm1.3} -> \bf{67.1_{\pm0.4}} | Helped |
| Include Sib&L-NDMV in the unweighted ensemble (ablation) | Our parse aggregation · WSJ test set | UAS: 67.9_{\pm0.1} -> {{67.9}}_{\pm0.3} | No clear effect |
| Include Sib&L-NDMV and weight individuals by validation performance (ablation) | Our parse aggregation · WSJ test set | UAS: 67.9_{\pm0.1} -> {\underline{68.4}}_{\pm0.4}^{*} | Helped |
| Use society entropy selection with Sib&L-NDMV included | Ensemble selection · WSJ test set | UAS: 67.9_{\pm0.1} -> \bf{68.8} | Helped |
| Use F1 rather than UAS as the aggregation objective (ablation) | Unweighted parse aggregation · WSJ test set | UAS: {{65.7}}_{\pm0.9} -> 59.2_{\pm2.1} | Hurt |
| Use F1 rather than UAS as the aggregation objective (ablation) | Weighted parse aggregation · WSJ test set | F1: 63.7_{\pm0.4} -> \bf{66.6}_{\pm0.8} | Helped |
Surprising findings
- Ensembling all individuals performs worse than a moderately sized ensemble because weak individuals accumulate errors.
- Adding Sib&L-NDMV to the ensemble does not improve over its own performance, showing limited robustness to weak components.
Limitations
- Ensemble selection is restricted to classification-style tasks, so it does not directly extend to other structured outputs.
- Aggregation only handles projective dependency parses.
- Requires a small annotated validation set (250 sentences) for selection and weighting.
Experimental setup
Datasets: Penn Treebank WSJ
Benchmarks: Penn Treebank WSJ test set
Metrics: unlabeled attachment score F1 inference time
Baselines: CRFAE NDMV NE-DMV L-NDMV Sib-NDMV Sib&L-NDMV CIM aggregation ensemble validation Kuncheva's diversity Disagreement PCDM
Builds on: Eisner algorithm DMV CRFAE Sib-NDMV customized Ising model
Closest papers by meaning
- Ensemble Distillation for Unsupervised Constituency Parsing (ICLR 2024 · similarity 0.88)
- Dependency Parsing is More Parameter-Efficient with Normalization (NeurIPS 2025 · similarity 0.84)
- Dynamic Syntactic Feature Filtering and Injecting Networks for Cross-lingual Dependency Parsing (AAAI 2025 · similarity 0.83)
- Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling (NeurIPS 2026 · similarity 0.82)
- Domain2Vec: Vectorizing Datasets to Find the Optimal Data Mixture without Training (ICML 2025 · similarity 0.81)
- Improving Unsupervised Constituency Parsing via Maximizing Semantic Information (ICLR 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.