Research map / Architectures and efficiency / Attention and transformers
Parsing as Pretraining
David Vilares, Michalina Strzyz, Anders Søgaard, Carlos Gómez-Rodríguez
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune BERT rather than freeze it (ablation) | Parsing as Pretraining with BERT and a feed-forward label predictor · PTB test | labeled bracketing F1: 78.2 -> 93.5 | Improved |
| Fine-tune BERT rather than freeze it (ablation) | Parsing as Pretraining with BERT and a feed-forward label predictor · EN-EWT UD test | LAS: 63.0 -> 78.8 | Improved |
| Use BERT instead of ELMo with the feed-forward parser and fine-tune the encoder (ablation) | Parsing as Pretraining · PTB test | labeled bracketing F1: 75.8 -> 93.5 | Improved |
| Use contextualized BERT vectors instead of random embeddings, with fine-tuning (ablation) | Parsing as Pretraining · EN-EWT UD test | LAS: 39.7 -> 78.8 | Improved |
| Add a 2-layer BiLSTM decoder (ablation) | Parsing as Pretraining with frozen BERT · PTB test | labeled bracketing F1: 78.2 -> 92.2 | Improved |
| Add a 2-layer BiLSTM decoder (ablation) | Parsing as Pretraining with fine-tuned BERT · PTB test | labeled bracketing F1: 93.5 -> 93.7 | No clear effect |
| Use pretrained word vectors instead of random embeddings (ablation) | Frozen feed-forward parser · PTB test | labeled bracketing F1: 32.9 -> 41.4 | Improved |
| Use a task-specific BiLSTM decoder (ablation) | Parsing as Pretraining with frozen input representations · PTB test | labeled bracketing F1: 78.2 -> 92.2; 41.4 -> 89.0 | Mixed |
Closest papers by meaning
- Discontinuous Constituent Parsing with Pointer Networks (AAAI 2020 · similarity 0.88)
- Differentiable Perturb-and-Parse: Semi-Supervised Parsing with a Structured Variational Autoencoder (ICLR 2019 · similarity 0.88)
- Learning the Wrong Lessons: Syntactic-Domain Spurious Correlations in Language Models (NeurIPS 2025 · similarity 0.86)
- Dependency Parsing is More Parameter-Efficient with Normalization (NeurIPS 2025 · similarity 0.86)
- Dynamic Syntactic Feature Filtering and Injecting Networks for Cross-lingual Dependency Parsing (AAAI 2025 · similarity 0.86)
- DEPT: Decoupled Embeddings for Pre-training Language Models (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
