Research map / Vision-language and multimodal / Human and embodied modeling
Visualizing and Measuring the Geometry of BERT
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, Been Kim
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply the trained semantic probe to BERT embeddings | BERT-base nearest-centroid WSD · WSD test suite | F1 score: 71.1 -> 71.5 | Improved |
| Use BERT context embeddings for nearest-centroid WSD instead of ELMo | WSD · WSD test suite | F1 score: 70.1 -> 71.1 | Improved |
| Use model-wide attention vectors as input to a linear dependency-existence classifier | BERT-base attention · Penn Treebank dependency existence | accuracy: 85.8% | Improved |
| Use model-wide attention vectors as input to a linear dependency-type classifier | BERT-base attention · Penn Treebank dependency relation type | accuracy: 71.9% | Improved |
| Reduce trained semantic-probe dimensionality from 768 to 512 (ablation) | BERT-base semantic probe · WSD test suite | accuracy: 71.26 -> 71.52 | Improved |
| Reduce trained semantic-probe dimensionality to 256 or 128 (ablation) | BERT-base semantic probe · WSD test suite | accuracy: 71.26; 71.29; 71.21 | No clear effect |
| Reduce trained semantic-probe dimensionality to 64, 32, or 16 (ablation) | BERT-base semantic probe · WSD test suite | accuracy: 70.19; 68.01; 65.34 | Hurt |
| Train the semantic probe rather than use a random probe of the same dimensionality (ablation) | BERT-base semantic probe · WSD test suite | accuracy: 71.26 vs 70.74; 71.52 vs 70.51; 71.29 vs 69.92; 71.21 vs 69.56; 70.19 vs 68.00; 68.01 vs 64.62; 65.34 vs 61.01 | Improved |
| Concatenate a sentence with a sentence using the same keyword in another sense | BERT-base context embeddings · SemCor sentence pairs | misclassification rate: 2.43% -> 8.23% | Hurt |
| Project keyword embeddings into the trained semantic subspace (ablation) | BERT-base context embeddings · SemCor sentence pairs | similarity ratio: 1.28 -> 1.58 (concatenated); 1.43 -> 1.88 (individual) | Improved |
| Project keyword embeddings into the trained semantic subspace (ablation) | BERT-base context embeddings · SemCor sentence pairs | misclassification rate: 7.31% vs 8.23% (concatenated); 2.27% vs 2.43% (individual) | Improved |
| Concatenate a random sentence instead of a different-sense sentence (ablation) | BERT-base context embeddings · SemCor sentence pairs | similarity ratio | No clear effect |
Closest papers by meaning
- Probing BERT in Hyperbolic Spaces (ICLR 2021 · similarity 0.87)
- Invariant Features in Language Models: Geometric Characterization and Model Attribution (NeurIPS 2026 · similarity 0.86)
- Emergence of a High-Dimensional Abstraction Phase in Language Transformers (ICLR 2025 · similarity 0.86)
- Geometric Factual Recall in Transformers (NeurIPS 2026 · similarity 0.85)
- Tracing the Representation Geometry of Language Models from Pretraining to Post-training (NeurIPS 2025 · similarity 0.85)
- A Polar coordinate system represents syntax in large language models (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
