Research map / Architectures and efficiency / Attention and transformers
What do you learn from context? Probing for sentence structure in contextualized word representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add contextual encoder representations rather than use lexical representations only (ablation) | BERT-large probing model · Macro Average | F1: 75.2 -> 87.3 | Helped |
| add contextual encoder representations rather than use lexical representations only (ablation) | BERT-large probing model · Winograd coref. | F1: 53.0 -> 61.4 | Helped |
| add contextual encoder representations rather than use lexical representations only (ablation) | BERT-large probing model · SPR1 | F1: 79.6 -> 85.8 | Helped |
| replace concatenation with learned scalar mixing across layers (ablation) | GPT probing model · Macro Average | F1: 83.2 -> 84.4 | Helped |
| replace concatenation with learned scalar mixing across layers (ablation) | BERT-base probing model · Macro Average | F1: 84.8 -> 86.3 | Helped |
| replace concatenation with learned scalar mixing across layers (ablation) | BERT-large probing model · Macro Average | F1: 84.2 -> 87.3 | Helped |
| use deeper BERT-large instead of BERT-base with scalar mixing | BERT probing model · Macro Average | F1: 86.3 -> 87.3 | Helped |
| use BERT-large instead of ELMo | contextual encoder probing · OntoNotes coref. | F1: 7.4 F1 points | Helped |
| use BERT-large instead of ELMo | contextual encoder probing · Winograd coref. | F1: 7.8 F1 points | Helped |
| train GPT on Billion Word Benchmark text rather than Books Corpus text (ablation) | GPT · Probing suite | F1: +0.15 F1 on average | No clear effect |
| use randomized orthonormal recurrent and projection weights rather than pretrained weights (ablation) | ELMo encoder · Probing suite | F1 | Hurt |
| add a width-3 CNN over local context to lexical representations (ablation) | Lexical baseline · Probing suite | F1: 72% | Helped |
Closest papers by meaning
- Quantifying the Plausibility of Context Reliance in Neural Machine Translation (ICLR 2024 · similarity 0.87)
- The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentation (NeurIPS 2025 · similarity 0.87)
- Less is More: Local Intrinsic Dimensions of Contextual Language Models (NeurIPS 2025 · similarity 0.87)
- Narrowing the Gap between Supervised and Unsupervised Sentence Representation Learning with Large Language Model (AAAI 2024 · similarity 0.87)
- ICLR: In-Context Learning of Representations (ICLR 2025 · similarity 0.86)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.