Research map / Architectures and efficiency / Attention and transformers
Dependency Parsing is More Parameter-Efficient with Normalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that normalizing biaffine scores (dividing by sqrt(d_k), as in Transformer attention) lets a stacked-BiLSTM dependency parser match or beat deeper unnormalized parsers with up to 85% fewer trained parameters, across semantic, syntactic, and latent graph tasks.
Problem
Biaffine-scored dependency parsers omit the 1/sqrt(d_k) score normalization used in Transformer attention, so high-variance pre-softmax scores produce sharp softmax outputs and the models appear to need extra BiLSTM layers (overparameterization) to compensate.
Approach
Argues theoretically (singular-value dynamics plus a variance-rank claim) that depth acts as implicit normalization of biaffine scores, and shows that explicit score scaling removes the need for extra BiLSTM layers, giving comparable or better parsers with far fewer parameters.
Key results
- With score scaling (a=1/sqrt(d)) and a single BiLSTM layer, the parser matches or beats the unnormalized three-layer Bhatt et al. baseline, e.g. ADE 0.668 vs 0.653 and CoNLL04 0.597 vs 0.566, with up to 85% fewer trained parameters.
- Best normalized model (N=3) reports ADE 0.686, CoNLL04 0.602, SciERC 0.320, ERFGC 0.708, beating the baseline's 0.653, 0.566, 0.257, 0.701 respectively, and sets a new ERFGC state of the art.
- On SciERC, normalizing scores without any BiLSTM layers yields a 23% performance increase with much lower variance (0.181 vs 0.147 at N=0 with the 1/sqrt(d) scaling).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale biaffine scores by 1/√d rather than use raw scores | Dependency parser, N=3 · ADE | micro-F1: 0.662 -> 0.686 | Helped |
| Scale biaffine scores by 1/√d rather than use raw scores | Dependency parser, N=3 · CoNLL04 | micro-F1: 0.562 -> 0.602 | Helped |
| Scale biaffine scores by 1/√d rather than use raw scores | Dependency parser, N=3 · SciERC | micro-F1: 0.299 -> 0.320 | Helped |
| Scale biaffine scores by 1/√d rather than use raw scores | Dependency parser, N=3 · ERFGC | micro-F1: 0.705 -> 0.708 | Helped |
| Scale biaffine scores by 1/√d rather than use raw scores | Dependency parser, N=3 · enEWT | LAS: 0.804 -> 0.807 | Helped |
| Scale biaffine scores by 1/√d rather than use raw scores | Dependency parser, N=3 · SciDTB | LAS: 0.915 -> 0.919 | Helped |
| Add score normalization to the parser (ablation) | Dependency parser, N=0 · SciERC | micro-F1: 0.147 -> 0.181 | Helped |
| Add both the tagger BiLSTM and tag embeddings (ablation) | Semantic dependency parser · ADE, CoNLL04, SciERC, ERFGC | mean micro-F1: 0.534 -> 0.541 | Helped |
| Remove tag embeddings (ablation) | Semantic dependency parser with tagger BiLSTM · ADE, CoNLL04, SciERC, ERFGC | mean micro-F1: 0.541 -> 0.517 | Hurt |
| Remove the tagger BiLSTM (ablation) | Semantic dependency parser with tag embeddings · ADE, CoNLL04, SciERC, ERFGC | mean micro-F1: 0.541 -> 0.514 | Hurt |
| Increase edge-projection MLP output dimension from 100 to 500 (ablation) | Parser with raw biaffine scores · SciERC | micro-F1: 0.302 -> 0.281 | Hurt |
| Increase edge-projection MLP output dimension from 100 to 500 (ablation) | Parser with normalized biaffine scores · ADE | micro-F1: 0.686 -> 0.685 | No clear effect |
Surprising findings
- Full fine-tuning with 100-400M parameters generally shows no top-performance gain from normalization, though normalization still speeds convergence for several models.
- Without normalization, adding BiLSTM layers makes training unstable at deeper stacks (e.g., N=6 needing longer training), and LayerNorm between BiLSTM layers often breaks convergence on CoNLL04 and SciERC.
Limitations
- ERFGC is not publicly available and was obtained from authors, limiting reproducibility; ACE04/ACE05 were excluded due to cost.
- Top performance gains from normalization vanish under full fine-tuning and at deeper BiLSTM stacks, so the benefit is conditional on architecture and training budget.
- Syntactic experiments use SciDTB parsed with Stanza rather than the standard Penn Treebank, and the Stanza-based annotations may be noisy.
Experimental setup
Datasets: ADE CoNLL04 SciERC ERFGC enEWT SciDTB Universal Dependencies (Arabic, Chinese, Italian, Japanese, Spanish, Wolof) PCQM-Contact CIFAR10 Superpixel QM9
Benchmarks: ADE CoNLL04 SciERC ERFGC enEWT SciDTB PCQM-Contact CIFAR10 Superpixel QM9
Metrics: micro-averaged F1 LAS UAS
Baselines: Bhatt et al. biaffine parser (unnormalized, N=3) Dozat and Manning biaffine parser
Builds on: Dozat and Manning biaffine parser Transformer scaled dot-product attention Bhatt et al. semantic dependency parser graph attention networks Chu-Liu/Edmonds MST
Closest papers by meaning
- Deep Biaffine Attention for Neural Dependency Parsing (ICLR 2017 · similarity 0.92)
- Global Greedy Dependency Parsing (AAAI 2020 · similarity 0.88)
- Attention-based Belief or Disbelief Feature Extraction for Dependency Parsing (AAAI 2018 · similarity 0.87)
- Dynamic Syntactic Feature Filtering and Injecting Networks for Cross-lingual Dependency Parsing (AAAI 2025 · similarity 0.86)
- Improving Unsupervised Constituency Parsing via Maximizing Semantic Information (ICLR 2025 · similarity 0.85)
- Punctuation-aware Hybrid Trainable Sparse Attention for Large Language Models (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.