Research map / Large language models / Language, safety and interpretability
Learning syntax without semantics: Disentangled tiny language models
Ezra Winston, J Zico Kolter
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train on relexicalized grammatical nonsense (SAMBAL) instead of original text | GPT-BERT 30M · BLiMP (long) | accuracy: 79.2 -> 79.3 | No clear effect |
| train on SAMBAL text | GPT-BERT 30M · EWoK | accuracy: 54.3 -> 50.8 | Hurt |
| train on SAMBAL text | GPT-BERT 30M · SyntaxGym (all suites, long) | accuracy: 70.47 -> 66.0 | Hurt |
| train on SAMBAL text | GPT-BERT 30M · SyntaxGym without reflexive suites | accuracy: 78.6 -> 79.8 | No clear effect |
| train on SAMBAL text | GPT-BERT 30M · grammar-plausibility conflict set | accuracy: 10.2% -> 100% | Improved |
| train on SAMBAL text | GPT-BERT 30M · swap probe semantic plausibility | accuracy: 100.0 -> 55.0 | Hurt |
| train 5M model on SAMBAL text at 10% data | 5M-parameter model · BLiMP | accuracy: 58.0 -> 60.6 | Improved |
| SAMBAL at 14M/30M models with 50-100% data | larger models · BLiMP | accuracy | No clear effect |
| 5M SAMBAL model vs 30M baseline at 10% data | 5M-parameter model · BLiMP | accuracy: 59.6 -> 60.6 | Improved |
| remove corpus-conditioned candidate sampling from SAMBAL pipeline (ablation) | SAMBAL-short · BLiMP | accuracy: 71.2 -> 66.42 | Hurt |
| ablate gender / humanness features in relexicalization (rather than preserve) (ablation) | SAMBAL-short · BLiMP | accuracy: 71.2 -> 70.47 (gender), 69.86 (humanness) | Hurt |
| LoRA fine-tune SAMBAL model on Lord of the Rings | SAMBAL (Long) · LoTR perplexity | perplexity: 190.4 -> 36.3 | Improved |
Closest papers by meaning
- Syntax vs. Semantics: How Transformers Learn Deep Dependencies (ICML 2026 · similarity 0.87)
- Learning the Wrong Lessons: Syntactic-Domain Spurious Correlations in Language Models (NeurIPS 2025 · similarity 0.87)
- Distinguishing the Knowable from the Unknowable with Language Models (ICML 2024 · similarity 0.87)
- From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and When (NeurIPS 2024 · similarity 0.86)
- Evaluating the Zero-shot Robustness of Instruction-tuned Language Models (ICLR 2024 · similarity 0.86)
- Finding the Minimal Parameter Budget for Implicit Reasoning: A Data Complexity Driven Scaling Law for Language Models (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
