Research map / Label-efficient and robust learning / Recognition and long-tail
Deep Clustering of Text Representations for Supervision-Free Probing of Syntax
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace KMeans on mBERT vectors with SAE compression (ablation) | POSI on Penn Treebank WSJ · Penn Treebank WSJ 45-tag POSI | M1 accuracy: 72.4 ($\pm$2.9) -> 75.3 ($\pm$1.4) | Helped |
| Fine-tune compressed representations using the clustering objective (ablation) | POSI on Penn Treebank WSJ · Penn Treebank WSJ 45-tag POSI | M1 accuracy: 75.3 ($\pm$1.4) -> 77.6 ($\pm$1.5) | Helped |
| Add morphological character features (ablation) | SyntDEC POSI · Penn Treebank WSJ 45-tag POSI | M1 accuracy: 77.6 ($\pm$1.5) -> 79.5 ($\pm$0.9) | Helped |
| Use SyntDEC rather than SAE (ablation) | POSI across 10 Universal Treebank languages · 12-tag Universal Treebank POSI mean | M1 accuracy: 70.9% to 75.7% | Helped |
| Use SyntDEC rather than the previous best reported method | POSI across 10 Universal Treebank languages · 12-tag Universal Treebank POSI mean | M1 accuracy: 71.4% to 75.7% | Helped |
| Use CBoW instead of SAE reconstruction with E-BERT (ablation) | SyntDEC POSI · Penn Treebank WSJ 45-tag POSI | M1 accuracy: 72.7 ($\pm$1.2) -> 74.4 ($\pm$0.6) | Helped |
| Use CBoW instead of SAE reconstruction with mBERT (ablation) | SyntDEC POSI · Penn Treebank WSJ 45-tag POSI | M1 accuracy: 77.8 ($\pm$1.4) -> 75.9 ($\pm$0.3) | Hurt |
| Add trailing trigram character embeddings rather than unigram embeddings (ablation) | SyntDEC-Morph POSI · Penn Treebank WSJ 45-tag POSI | M1 accuracy: 77.9 ($\pm$2.3) -> 79.5 ($\pm$0.9) | Helped |
| Add trailing trigram character embeddings rather than bigram embeddings (ablation) | SyntDEC-Morph POSI · Penn Treebank WSJ 45-tag POSI | M1 accuracy: 77.4 ($\pm$2.4) -> 79.5 ($\pm$0.9) | Helped |
| Represent spans by concatenating first and last token rather than mean pooling (ablation) | SyntDEC CoLab · CoLab WSJ test | F1μ: 40.8 ($\pm$1.1) -> 64.0 ($\pm$0.4) | Helped |
| Fine-tune SAE span representations with SyntDEC clustering (ablation) | CoLab with concatenated span endpoints · CoLab WSJ test | F1μ: 61.2 ($\pm$1.2) -> 64.0 ($\pm$0.4) | Helped |
| Use multilingual BERT rather than English BERT representations (ablation) | SyntDEC CoLab · CoLab WSJ test | F1μ: 60.8 ($\pm$0.7) -> 64.0 ($\pm$0.4) | Helped |
Closest papers by meaning
- MMTEB: Massive Multilingual Text Embedding Benchmark (ICLR 2025 · similarity 0.87)
- Ensemble Distillation for Unsupervised Constituency Parsing (ICLR 2024 · similarity 0.86)
- A Polar coordinate system represents syntax in large language models (NeurIPS 2024 · similarity 0.86)
- Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe (NeurIPS 2024 · similarity 0.85)
- Improving Unsupervised Constituency Parsing via Maximizing Semantic Information (ICLR 2025 · similarity 0.85)
- Data Mixture Inference Attack: BPE Tokenizers Reveal Training Data Compositions (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.