Research map / Large language models / Language, safety and interpretability
Semantics-Aware BERT for Language Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add explicit SRL-derived contextual semantics | BERT_LARGE · SQuAD 2.0 test | F1: 83.6 -> 85.2 | Helped |
| Add explicit SRL-derived contextual semantics | BERT_LARGE · SNLI test | accuracy: 91.1 -> 91.6 | Helped |
| Add explicit SRL-derived contextual semantics | BERT_LARGE · GLUE | RTE accuracy: 70.1 -> 84.5 | Helped |
| Add explicit SRL-derived contextual semantics | BERT_LARGE · GLUE | SST-2 accuracy: 94.9 -> 94.6 | Hurt |
| Directly concatenate SRL embeddings with BERT subword embeddings instead of word-aligning and integrating them (ablation) | BERT_LARGE · SNLI dev | accuracy: 91.3 -> 91.5 | Helped |
| Use SemBERT's word-aligned semantic integration rather than direct subword-level SRL concatenation (ablation) | BERT_LARGE+SRL · SNLI dev | accuracy: 91.5 -> 92.3 | Helped |
| Use SemBERT's word-aligned semantic integration rather than direct subword-level SRL concatenation (ablation) | BERT_LARGE+SRL · SQuAD 2.0 dev | F1: 83.1 -> 83.6 | Helped |
| Vary the maximum number of predicate-argument structures from 1 to 5 (ablation) | SemBERT · SNLI | accuracy: 91.49; 91.36; 91.57; 91.29; 91.42 | Mixed |
| Corrupt 20% of SRL labels (ablation) | SemBERT · SQuAD 2.0 | F1: 87.93 -> 87.31 | Hurt |
| Corrupt 40% of SRL labels (ablation) | SemBERT · SQuAD 2.0 | F1: 87.93 -> 87.24 | Hurt |
| Add synthetic self-training data augmentation | SemBERT_LARGE · SQuAD 2.0 test | F1: 85.2 -> 87.9 | Helped |
Closest papers by meaning
- Incorporating BERT into Neural Machine Translation (ICLR 2020 · similarity 0.88)
- SensEmBERT: Context-Enhanced Sense Embeddings for Multilingual Word Sense Disambiguation (AAAI 2020 · similarity 0.87)
- Probing Natural Language Inference Models through Semantic Fragments (AAAI 2020 · similarity 0.87)
- What do you learn from context? Probing for sentence structure in contextualized word representations (ICLR 2019 · similarity 0.87)
- Semantics-Aware Inferential Network for Natural Language Understanding (AAAI 2021 · similarity 0.86)
- K-BERT: Enabling Language Representation with Knowledge Graph (AAAI 2020 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.